Server keeps crashing in last 3 days - Swap issue?

heusdens · Nov 14, 2013

Hi

I've had server uptime of 3 months with no problems. In the past 3 days however it started to crash at around 3am everyday, had to reboot. Nothing else changed except WHM was upgraded to WHM 11.40.0. When looking at messages log I see the following similar logs for each day:

Code:

Nov 15 03:17:23 server1 kernel: Out of memory: Kill process 26830 (php) score 6 or sacrifice child
Nov 15 03:17:23 server1 kernel: Killed process 26830, UID 505, (php) total-vm:252320kB, anon-rss:6156kB, file-rss:52kB
Nov 15 03:17:23 server1 kernel: php invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0, oom_score_adj=0
Nov 15 03:17:23 server1 kernel: php cpuset=/ mems_allowed=0
Nov 15 03:17:23 server1 kernel: Pid: 28505, comm: php Not tainted 2.6.32-220.4.2.el6.x86_64 #1
Nov 15 03:17:23 server1 kernel: Call Trace:
Nov 15 03:17:23 server1 kernel: [<ffffffff810c2b51>] ? cpuset_print_task_mems_allowed+0x91/0xb0
Nov 15 03:17:23 server1 kernel: [<ffffffff811138d0>] ? dump_header+0x90/0x1b0
Nov 15 03:17:23 server1 kernel: [<ffffffff8120d69c>] ? security_real_capable_noaudit+0x3c/0x70
Nov 15 03:17:23 server1 kernel: [<ffffffff81113d5a>] ? oom_kill_process+0x8a/0x2c0
Nov 15 03:17:23 server1 kernel: [<ffffffff81113c91>] ? select_bad_process+0xe1/0x120
Nov 15 03:17:23 server1 kernel: [<ffffffff811141b0>] ? out_of_memory+0x220/0x3c0
Nov 15 03:17:23 server1 kernel: [<ffffffff81123ece>] ? __alloc_pages_nodemask+0x89e/0x940
Nov 15 03:17:23 server1 kernel: [<ffffffff81158a1a>] ? alloc_pages_current+0xaa/0x110
Nov 15 03:17:23 server1 kernel: [<ffffffff81110cf7>] ? __page_cache_alloc+0x87/0x90
Nov 15 03:17:23 server1 kernel: [<ffffffff8112691b>] ? __do_page_cache_readahead+0xdb/0x210
Nov 15 03:17:23 server1 kernel: [<ffffffff81126a71>] ? ra_submit+0x21/0x30
Nov 15 03:17:23 server1 kernel: [<ffffffff81111fc3>] ? filemap_fault+0x4c3/0x500
Nov 15 03:17:23 server1 kernel: [<ffffffff8113b164>] ? __do_fault+0x54/0x510
Nov 15 03:17:23 server1 kernel: [<ffffffff8113b717>] ? handle_pte_fault+0xf7/0xb50
Nov 15 03:17:23 server1 kernel: [<ffffffff81158a1a>] ? alloc_pages_current+0xaa/0x110
Nov 15 03:17:23 server1 kernel: [<ffffffff810471c7>] ? pte_alloc_one+0x37/0x50
Nov 15 03:17:23 server1 kernel: [<ffffffff8113c354>] ? handle_mm_fault+0x1e4/0x2b0
Nov 15 03:17:23 server1 kernel: [<ffffffff81042b79>] ? __do_page_fault+0x139/0x480
Nov 15 03:17:23 server1 kernel: [<ffffffff811423da>] ? do_mmap_pgoff+0x33a/0x380
Nov 15 03:17:23 server1 kernel: [<ffffffff814f23fe>] ? do_page_fault+0x3e/0xa0
Nov 15 03:17:23 server1 kernel: [<ffffffff814ef7b5>] ? page_fault+0x25/0x30
Nov 15 03:17:23 server1 kernel: Mem-Info:
Nov 15 03:17:23 server1 kernel: Node 0 DMA per-cpu:
Nov 15 03:17:23 server1 kernel: CPU    0: hi:    0, btch:   1 usd:   0
Nov 15 03:17:23 server1 kernel: CPU    1: hi:    0, btch:   1 usd:   0
Nov 15 03:17:23 server1 kernel: Node 0 DMA32 per-cpu:
Nov 15 03:17:23 server1 kernel: CPU    0: hi:  186, btch:  31 usd:  77
Nov 15 03:17:23 server1 kernel: CPU    1: hi:  186, btch:  31 usd:  23
Nov 15 03:17:23 server1 kernel: active_anon:286932 inactive_anon:95910 isolated_anon:15173
Nov 15 03:17:23 server1 kernel: active_file:235 inactive_file:423 isolated_file:0
Nov 15 03:17:23 server1 kernel: unevictable:0 dirty:0 writeback:657 unstable:0
Nov 15 03:17:23 server1 kernel: free:13264 slab_reclaimable:5243 slab_unreclaimable:25682
Nov 15 03:17:23 server1 kernel: mapped:230 shmem:42 pagetables:53981 bounce:0
Nov 15 03:17:23 server1 kernel: Node 0 DMA free:8324kB min:332kB low:412kB high:496kB active_anon:1424kB inactive_anon:1480kB active_file:0kB inactive_file:44kB unevictable:0kB isolated(anon):384kB isolated(file):0kB present:15364kB mlocked:0kB dirty:0kB writeback:28kB mapped:16kB shmem:0kB slab_reclaimable:12kB slab_unreclaimable:1100kB kernel_stack:800kB pagetables:1540kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:285 all_unreclaimable? no
Nov 15 03:17:23 server1 kernel: lowmem_reserve[]: 0 2004 2004 2004
Nov 15 03:17:23 server1 kernel: Node 0 DMA32 free:44732kB min:44720kB low:55900kB high:67080kB active_anon:1146304kB inactive_anon:382160kB active_file:940kB inactive_file:1648kB unevictable:0kB isolated(anon):60308kB isolated(file):0kB present:2052308kB mlocked:0kB dirty:0kB writeback:2600kB mapped:904kB shmem:168kB slab_reclaimable:20960kB slab_unreclaimable:101628kB kernel_stack:8736kB pagetables:214384kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:7872 all_unreclaimable? no
Nov 15 03:17:23 server1 kernel: lowmem_reserve[]: 0 0 0 0
Nov 15 03:17:23 server1 kernel: Node 0 DMA: 22*4kB 104*8kB 102*16kB 25*32kB 4*64kB 1*128kB 4*256kB 3*512kB 0*1024kB 1*2048kB 0*4096kB = 8344kB
Nov 15 03:17:23 server1 kernel: Node 0 DMA32: 10159*4kB 4*8kB 2*16kB 2*32kB 0*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 44732kB
Nov 15 03:17:23 server1 kernel: 28420 total pagecache pages
Nov 15 03:17:23 server1 kernel: 27672 pages in swap cache
Nov 15 03:17:23 server1 kernel: Swap cache stats: add 3649062, delete 3621390, find 515396/799151
Nov 15 03:17:23 server1 kernel: Free swap  = 0kB
Nov 15 03:17:23 server1 kernel: Total swap = 4194296kB
Nov 15 03:17:23 server1 kernel: 524284 pages RAM
Nov 15 03:17:23 server1 kernel: 10550 pages reserved
Nov 15 03:17:23 server1 kernel: 44773 pages shared
Nov 15 03:17:23 server1 kernel: 469818 pages non-shared
Nov 15 03:17:23 server1 kernel: [ pid ]   uid  tgid total_vm      rss cpu oom_adj oom_score_adj name
Nov 15 03:17:23 server1 kernel: [  553]     0   553     2791        1   1     -17         -1000 udevd
Nov 15 03:17:23 server1 kernel: [ 1322]     0  1322     2279        1   1       0             0 dhclient
Nov 15 03:17:23 server1 kernel: [ 1360]     0  1360    23298       40   0     -17         -1000 auditd
Nov 15 03:17:23 server1 kernel: [ 1411]     0  1411     1539        1   1       0             0 portreserve
Nov 15 03:17:23 server1 kernel: [ 1418]     0  1418    62270      362   1       0             0 rsyslogd
Nov 15 03:17:23 server1 kernel: [ 1447]     0  1447     2704       27   1       0             0 irqbalance
Nov 15 03:17:23 server1 kernel: [ 1457]    25  1457    59401      290   1       0             0 named
Nov 15 03:17:23 server1 kernel: [ 1478]    32  1478     4743       16   1       0             0 rpcbind
Nov 15 03:17:23 server1 kernel: [ 1516]    81  1516     7911        1   1       0             0 dbus-daemon
Nov 15 03:17:23 server1 kernel: [ 1546]     0  1546     1019        0   0       0             0 acpid
Nov 15 03:17:23 server1 kernel: [ 1555]    68  1555     6263       84   0       0             0 hald
Nov 15 03:17:23 server1 kernel: [ 1556]     0  1556     4526        1   0       0             0 hald-runner
Nov 15 03:17:23 server1 kernel: [ 1584]     0  1584     5055        2   1       0             0 hald-addon-inpu
Nov 15 03:17:23 server1 kernel: [ 1599]    68  1599     4451        1   0       0             0 hald-addon-acpi
Nov 15 03:17:23 server1 kernel: [ 1615]     0  1615   113066       43   1       0             0 automount
Nov 15 03:17:23 server1 kernel: [ 1643]     0  1643     1691        0   1       0             0 mcelog
Nov 15 03:17:23 server1 kernel: [ 1655]     0  1655    16563        0   1     -17         -1000 sshd
Nov 15 03:17:23 server1 kernel: [ 1909]    47  1909    17363       39   1       0             0 exim
Nov 15 03:17:23 server1 kernel: [ 1943]     0  1943    41599      289   0       0             0 /usr/local/cpan
Nov 15 03:17:23 server1 kernel: [ 1992]     0  1992    27543        1   0       0             0 abrtd
Nov 15 03:17:23 server1 kernel: [ 2005]     0  2005    27103       99   1       0             0 abrt-dump-oops
Nov 15 03:17:23 server1 kernel: [ 2018]     0  2018     4889       36   0       0             0 dovecot
Nov 15 03:17:23 server1 kernel: [ 2024]   495  2024    10457       36   1       0             0 pop3-login
Nov 15 03:17:23 server1 kernel: [ 2025]   495  2025    10457       35   1       0             0 pop3-login
Nov 15 03:17:23 server1 kernel: [ 2026]   495  2026    10458        1   1       0             0 imap-login
Nov 15 03:17:23 server1 kernel: [ 2027]   495  2027    10458        1   0       0             0 imap-login
Nov 15 03:17:23 server1 kernel: [ 2028]    97  2028     3315       19   1       0             0 anvil
Nov 15 03:17:23 server1 kernel: [ 2029]     0  2029     3347       35   0       0             0 log
Nov 15 03:17:23 server1 kernel: [ 2031]     0  2031     4092       23   1       0             0 config
Nov 15 03:17:23 server1 kernel: [ 2047]     0  2047    19541      130   1       0             0 httpd
Nov 15 03:17:23 server1 kernel: [ 2057]     0  2057    34676        0   0       0             0 pure-ftpd
Nov 15 03:17:23 server1 kernel: [ 2059]     0  2059    34048        1   0       0             0 pure-authd
Nov 15 03:17:23 server1 kernel: [ 2077]     0  2077    41599      240   1       0             0 spamd child
Nov 15 03:17:23 server1 kernel: [ 2078]   498  2078    83251       80   1       0             0 qpidd
Nov 15 03:17:23 server1 kernel: [ 2253]     0  2253    29319       19   1       0             0 crond
Nov 15 03:17:23 server1 kernel: [ 2264]     0  2264     5373        0   0       0             0 atd
Nov 15 03:17:23 server1 kernel: [ 2399]     0  2399     7713       81   0       0             0 cPhulkd - proce
Nov 15 03:17:23 server1 kernel: [ 2421]     0  2421    24488      157   0       0             0 cpsrvd-ssl
Nov 15 03:17:23 server1 kernel: [ 2433]     0  2433    25574       72   0       0             0 cpdavd - accept
Nov 15 03:17:23 server1 kernel: [ 2446]     0  2446     8269      118   1       0             0 queueprocd - qu
Nov 15 03:17:23 server1 kernel: [ 2528]     0  2528    12296      249   0       0             0 tailwatchd
Nov 15 03:17:23 server1 kernel: [ 2559]     0  2559     6768        4   1       0             0 cpanellogd - wa
Nov 15 03:17:23 server1 kernel: [ 2614]     0  2614     1015        1   1       0             0 mingetty
Nov 15 03:17:23 server1 kernel: [ 2616]     0  2616     1015        1   0       0             0 mingetty
Nov 15 03:17:23 server1 kernel: [ 2618]     0  2618     1015        1   0       0             0 mingetty
Nov 15 03:17:23 server1 kernel: [ 2620]     0  2620     1015        1   0       0             0 mingetty
Nov 15 03:17:23 server1 kernel: [ 2622]     0  2622     1015        1   0       0             0 mingetty
Nov 15 03:17:23 server1 kernel: [ 2624]     0  2624     1015        1   0       0             0 mingetty
Nov 15 03:17:23 server1 kernel: [ 2630]     0  2630     3087        1   1     -17         -1000 udevd
Nov 15 03:17:23 server1 kernel: [ 2631]     0  2631     3087        1   1     -17         -1000 udevd
Nov 15 03:17:23 server1 kernel: [ 2871]   496  2871    88497      178   1       0             0 memcached
Nov 15 03:17:23 server1 kernel: [ 7823]     0  7823    36234      117   0       0             0 /usr/local/cpan
Nov 15 03:17:23 server1 kernel: [ 8161]     0  8161      391       14   0       0             0 splitlogs
Nov 15 03:17:23 server1 kernel: [ 8162]     0  8162      391       14   1       0             0 splitlogs
Nov 15 03:17:23 server1 kernel: [ 8163]    99  8163    19541       61   1       0             0 httpd
Nov 15 03:17:23 server1 kernel: [ 8164]     0  8164    12674        1   1       0             0 leechprotect
Nov 15 03:17:23 server1 kernel: [ 8170]     0  8170    19541       69   0       0             0 httpd
Nov 15 03:17:23 server1 kernel: [ 8171]    99  8171   334493     1499   1       0             0 httpd
Nov 15 03:17:23 server1 kernel: [ 8172]    99  8172   334536     1495   0       0             0 httpd
Nov 15 03:17:23 server1 kernel: [ 8173]    99  8173   334567     1470   0       0             0 httpd
Nov 15 03:17:23 server1 kernel: [ 8298]    99  8298   334505     1554   1       0             0 httpd
Nov 15 03:17:23 server1 kernel: [27765]    99 27765   334585     1595   0       0             0 httpd
Nov 15 03:17:23 server1 kernel: [27818]    99 27818   334557     1406   0       0             0 httpd
Nov 15 03:17:23 server1 kernel: [  730]     0   730     2835       26   1       0             0 mysqld_safe
Nov 15 03:17:23 server1 kernel: [26144]    99 26144   334583     1443   0       0             0 httpd
Nov 15 03:17:23 server1 kernel: [26833]   505 26833    62562     1761   1       0             0 php
Nov 15 03:17:23 server1 kernel: [26836]   505 26836    62562     1608   1       0             0 php
Nov 15 03:17:23 server1 kernel: [26837]   505 26837    62562     1859   0       0             0 php
Nov 15 03:17:23 server1 kernel: [26840]   505 26840    62562     1062   0       0             0 php
Nov 15 03:17:23 server1 kernel: [26841]   505 26841    62562     2243   1       0             0 php
Nov 15 03:17:23 server1 kernel: [26842]   505 26842    62562     1605   0       0             0 php
Nov 15 03:17:23 server1 kernel: [26850]   505 26850    62562     1217   0       0             0 php
Nov 15 03:17:23 server1 kernel: [26852]   505 26852    62562     1043   0       0             0 php
Nov 15 03:17:23 server1 kernel: [26854]   505 26854    62562     1078   0       0             0 php
Nov 15 03:17:23 server1 kernel: [26855]   505 26855    61344     1025   1       0             0 php
Nov 15 03:17:23 server1 kernel: [26856]   505 26856    61344     1027   0       0             0 php
Nov 15 03:17:23 server1 kernel: [26857]   505 26857    61344     1038   0       0             0 php
Nov 15 03:17:23 server1 kernel: [26859]   505 26859    61344      751   1       0             0 php
Nov 15 03:17:23 server1 kernel: [26863]   505 26863    61344     1200   0       0             0 php
Nov 15 03:17:23 server1 kernel: [26867]   505 26867    61344     1327   0       0             0 php
Nov 15 03:17:23 server1 kernel: [26874]   505 26874    61344     1220   1       0             0 php
Nov 15 03:17:23 server1 kernel: [26877]   505 26877    61347     1353   1       0             0 php
Nov 15 03:17:23 server1 kernel: [26878]   505 26878    63080     1091   0       0             0 php
Nov 15 03:17:23 server1 kernel: [26880]   505 26880    61344     1083   0       0             0 php
Nov 15 03:17:23 server1 kernel: [26881]   505 26881    81711     1621   0       0             0 php
Nov 15 03:17:23 server1 kernel: [26883]   505 26883    62562     1108   0       0             0 php
Nov 15 03:17:23 server1 kernel: [26885]   505 26885    61344      942   1       0             0 php
Nov 15 03:17:23 server1 kernel: [26886]   505 26886    61344     1046   0       0             0 php
Nov 15 03:17:23 server1 kernel: [26887]   505 26887    61344     1011   1       0             0 php
Nov 15 03:17:23 server1 kernel: [26890]   505 26890    61344      986   0       0             0 php
Nov 15 03:17:23 server1 kernel: [26895]   505 26895    61344     1033   1       0             0 php
Nov 15 03:17:23 server1 kernel: [26898]   505 26898    61344     1547   1       0             0 php
Nov 15 03:17:23 server1 kernel: [26901]   505 26901    61344     1051   0       0             0 php
Nov 15 03:17:23 server1 kernel: [26902]   505 26902    61344      991   0       0             0 php
Nov 15 03:17:23 server1 kernel: [26903]   505 26903    62560      798   1       0             0 php
Nov 15 03:17:23 server1 kernel: [26904]   505 26904    61278     1038   1       0             0 php
Nov 15 03:17:23 server1 kernel: [26905]   505 26905    62562     1486   0       0             0 php
Nov 15 03:17:23 server1 kernel: [26909]   505 26909    61344     1177   1       0             0 php
Nov 15 03:17:23 server1 kernel: [26910]   505 26910    61344     1153   0       0             0 php
Nov 15 03:17:23 server1 kernel: [26916]   505 26916    61344      952   0       0             0 php
Nov 15 03:17:23 server1 kernel: [26918]   505 26918    61344      913   1       0             0 php
Nov 15 03:17:23 server1 kernel: [26919]   505 26919    61278     1270   0       0             0 php
Nov 15 03:17:23 server1 kernel: [26921]   505 26921    61344     1204   1       0             0 php
Nov 15 03:17:23 server1 kernel: [26922]   505 26922    61344      953   0       0             0 php
Nov 15 03:17:23 server1 kernel: [26923]   505 26923    61278     1173   0       0             0 php
Nov 15 03:17:23 server1 kernel: [26925]   505 26925    61344     1309   1       0             0 php
Nov 15 03:17:23 server1 kernel: [26926]   505 26926    61278      957   1       0             0 php
Nov 15 03:17:23 server1 kernel: [26927]   505 26927    61278     1406   0       0             0 php
Nov 15 03:17:23 server1 kernel: [26929]   505 26929    62560     1426   0       0             0 php
Nov 15 03:17:23 server1 kernel: [26930]   505 26930    61344     1287   0       0             0 php
Nov 15 03:17:23 server1 kernel: [26931]   505 26931    61344     1068   1       0             0 php
Nov 15 03:17:23 server1 kernel: [26932]   505 26932    61278     1226   0       0             0 php
Nov 15 03:17:23 server1 kernel: [26933]   505 26933    61342     1318   0       0             0 php
Nov 15 03:17:23 server1 kernel: [26936]   505 26936    61276      585   0       0             0 php
Nov 15 03:17:23 server1 kernel: [26937]   505 26937    61344     1341   0       0             0 php
Nov 15 03:17:23 server1 kernel: [26938]   505 26938    61344     1278   0       0             0 php
Nov 15 03:17:23 server1 kernel: [26939]   505 26939    61344     1241   1       0             0 php
Nov 15 03:17:23 server1 kernel: [26940]   505 26940    61344     1223   1       0             0 php
Nov 15 03:17:23 server1 kernel: [26941]   505 26941    61344      909   0       0             0 php
Nov 15 03:17:23 server1 kernel: [26969]   505 26969    62562     1550   1       0             0 php
Nov 15 03:17:23 server1 kernel: [26971]   505 26971    61342     1172   1       0             0 php
Nov 15 03:17:23 server1 kernel: [26977]   505 26977    62560      929   0       0             0 php
Nov 15 03:17:23 server1 kernel: [26980]   505 26980    61342     1145   0       0             0 php
Nov 15 03:17:23 server1 kernel: [26981]   505 26981    61342     1199   1       0             0 php
Nov 15 03:17:23 server1 kernel: [26982]   505 26982    62562      935   0       0             0 php
Nov 15 03:17:23 server1 kernel: [26983]   505 26983    61342     1197   0       0             0 php
Nov 15 03:17:23 server1 kernel: [26984]   505 26984    61344     1352   1       0             0 php
Nov 15 03:17:23 server1 kernel: [26985]   505 26985    61342     1108   1       0             0 php
Nov 15 03:17:23 server1 kernel: [26986]   505 26986    61342     1160   1       0             0 php
Nov 15 03:17:23 server1 kernel: [26987]   505 26987    61342      826   0       0             0 php
Nov 15 03:17:23 server1 kernel: [26988]   505 26988    61342     1036   0       0             0 php
Nov 15 03:17:23 server1 kernel: [26989]   505 26989    61342     1257   0       0             0 php
Nov 15 03:17:23 server1 kernel: [26991]   505 26991    61342      878   1       0             0 php
Nov 15 03:17:23 server1 kernel: [26992]   505 26992    61342      881   0       0             0 php
Nov 15 03:17:23 server1 kernel: [26993]    99 26993   334516     1309   1       0             0 httpd
Nov 15 03:17:23 server1 kernel: [26994]   505 26994    61344     1120   1       0             0 php
Nov 15 03:17:23 server1 kernel: [26995]   505 26995    61342     1039   1       0             0 php
Nov 15 03:17:23 server1 kernel: [26996]   505 26996    61214     1066   0       0             0 php
Nov 15 03:17:23 server1 kernel: [26998]   505 26998    63080      858   1       0             0 php
Nov 15 03:17:23 server1 kernel: [26999]   505 26999    62562      874   1       0             0 php
Nov 15 03:17:23 server1 kernel: [27000]   505 27000    62562     1605   1       0             0 php
Nov 15 03:17:23 server1 kernel: [27001]   505 27001    81793      960   0       0             0 php
Nov 15 03:17:23 server1 kernel: [27002]   505 27002    61344      956   1       0             0 php
Nov 15 03:17:23 server1 kernel: [27003]   505 27003    62560      892   1       0             0 php
Nov 15 03:17:23 server1 kernel: [27004]   505 27004    61342     1092   0       0             0 php
Nov 15 03:17:23 server1 kernel: [27005]   504 27005    62690     1954   0       0             0 php
Nov 15 03:17:23 server1 kernel: [27006]    99 27006   334448     1274   0       0             0 httpd
Nov 15 03:17:23 server1 kernel: [27007]   505 27007    61344      994   1       0             0 php
Nov 15 03:17:23 server1 kernel: [27008]   505 27008    62562      730   0       0             0 php
Nov 15 03:17:23 server1 kernel: [27011]    99 27011   334516     1434   0       0             0 httpd
Nov 15 03:17:23 server1 kernel: [27064]   505 27064    62634     1067   0       0             0 php
Nov 15 03:17:23 server1 kernel: [27069]    99 27069   334491     1305   1       0             0 httpd
Nov 15 03:17:23 server1 kernel: [27113]   505 27113    62560     1693   1       0             0 php
Nov 15 03:17:23 server1 kernel: [27147]   505 27147    61342     1010   0       0             0 php
Nov 15 03:17:23 server1 kernel: [27150]   493 27150    35554       44   1       0             0 crond
Nov 15 03:17:23 server1 kernel: [27152]   505 27152    61342      793   1       0             0 php
Nov 15 03:17:23 server1 kernel: [27160]   505 27160    63080     1188   0       0             0 php
Nov 15 03:17:23 server1 kernel: [27161]   505 27161    61344      922   0       0             0 php
Nov 15 03:17:23 server1 kernel: [27162]   505 27162    61342     1097   0       0             0 php
Nov 15 03:17:23 server1 kernel: [27169]   505 27169    61278      838   0       0             0 php
Nov 15 03:17:23 server1 kernel: [27172]   505 27172    61342      755   0       0             0 php
Nov 15 03:17:23 server1 kernel: [27173]   505 27173    81711     1710   1       0             0 php
Nov 15 03:17:23 server1 kernel: [27174]   505 27174    61342     1010   0       0             0 php
Nov 15 03:17:23 server1 kernel: [27175]   505 27175    62562     1331   0       0             0 php
Nov 15 03:17:23 server1 kernel: [27177]   505 27177    81505     1516   1       0             0 php
Nov 15 03:17:23 server1 kernel: [27180]   505 27180    80970      796   1       0             0 php
Nov 15 03:17:23 server1 kernel: [27181]   505 27181    61342      784   1       0             0 php
Nov 15 03:17:23 server1 kernel: [27183]   505 27183    80970      856   1       0             0 php
Nov 15 03:17:23 server1 kernel: [27187]   505 27187    80061      975   1       0             0 php
Nov 15 03:17:23 server1 kernel: [27214]   504 27214    62690     1567   0       0             0 php
Nov 15 03:17:23 server1 kernel: [27219]   504 27219    62492     1887   1       0             0 php
Nov 15 03:17:23 server1 kernel: [27222]   505 27222    81351     1312   1       0             0 php
Nov 15 03:17:23 server1 kernel: [27233]   504 27233    62492     1418   0       0             0 php
Nov 15 03:17:23 server1 kernel: [27246]   505 27246    81115      744   1       0             0 php
Nov 15 03:17:23 server1 kernel: [27249]     0 27249    13486      441   0       0             0 cpanellogd - wa
Nov 15 03:17:23 server1 kernel: [27284]   505 27284    62570      901   1       0             0 php
Nov 15 03:17:23 server1 kernel: [27325]   504 27325    55912     1141   1       0             0 php
Nov 15 03:17:23 server1 kernel: [27328]   505 27328    81792     1326   1       0             0 php
Nov 15 03:17:23 server1 kernel: [27329]   505 27329    55851      807   1       0             0 php
Nov 15 03:17:23 server1 kernel: [27334]     0 27334    14440      247   0       0             0 tailwatchd - ch
Nov 15 03:17:23 server1 kernel: [27337]   505 27337    61084     2478   1       0             0 php
Nov 15 03:17:23 server1 kernel: [27338]   504 27338    56403     1250   0       0             0 php
Nov 15 03:17:23 server1 kernel: [27341]   505 27341    55333      608   0       0             0 php
Nov 15 03:17:23 server1 kernel: [27354]   505 27354    61344     1958   0       0             0 php
Nov 15 03:17:23 server1 kernel: [27355]   505 27355    61344     1954   1       0             0 php
Nov 15 03:17:23 server1 kernel: [27361]   505 27361    55333      727   1       0             0 php
Nov 15 03:17:23 server1 kernel: [27364]   505 27364    55333      767   0       0             0 php
Nov 15 03:17:23 server1 kernel: [27365]   505 27365    55851      854   0       0             0 php
Nov 15 03:17:23 server1 kernel: [27366]   505 27366    55333      639   0       0             0 php
Nov 15 03:17:23 server1 kernel: [27380]   505 27380    80705     1194   0       0             0 php
Nov 15 03:17:23 server1 kernel: [27381]   505 27381    55851      880   0       0             0 php
Nov 15 03:17:23 server1 kernel: [27384]   505 27384    80188     1010   1       0             0 php
Nov 15 03:17:23 server1 kernel: [27385]   505 27385    81225     1175   1       0             0 php
Nov 15 03:17:23 server1 kernel: [27393]   505 27393    55657      481   1       0             0 php
Nov 15 03:17:23 server1 kernel: [27394]   505 27394    55333      697   0       0             0 php
Nov 15 03:17:23 server1 kernel: [27395]   505 27395    55399      808   0       0             0 php
Nov 15 03:17:23 server1 kernel: [27396]   505 27396    55333      887   0       0             0 php
Nov 15 03:17:23 server1 kernel: [27397]   505 27397    55527      616   1       0             0 php
Nov 15 03:17:23 server1 kernel: [27402]   505 27402    55399      714   0       0             0 php
Nov 15 03:17:23 server1 kernel: [27410]   505 27410    80736     1112   0       0             0 php
Nov 15 03:17:23 server1 kernel: [27411]   505 27411    61406     1696   1       0             0 php
Nov 15 03:17:23 server1 kernel: [27412]   505 27412    61279     3101   0       0             0 php
Nov 15 03:17:23 server1 kernel: [27413]   505 27413    61344     3191   0       0             0 php
Nov 15 03:17:23 server1 kernel: [27415]   505 27415    61278     2659   0       0             0 php
Nov 15 03:17:23 server1 kernel: [27416]   504 27416    62494     3248   1       0             0 php
Nov 15 03:17:23 server1 kernel: [27418]   505 27418    55333      732   1       0             0 php
Nov 15 03:17:23 server1 kernel: [27420]   505 27420    62562     3066   0       0             0 php
Nov 15 03:17:23 server1 kernel: [27422]   505 27422    55399      829   1       0             0 php
Nov 15 03:17:23 server1 kernel: [27424]   505 27424    61082     3052   1       0             0 php
Nov 15 03:17:23 server1 kernel: [27426]   504 27426    56403     1096   0       0             0 php
Nov 15 03:17:23 server1 kernel: [27437]   505 27437    81591      738   1       0             0 php
Nov 15 03:17:23 server1 kernel: [27442]   505 27442    63088     3414   1       0             0 php
Nov 15 03:17:23 server1 kernel: [27443]   505 27443    81544     1267   1       0             0 php
Nov 15 03:17:23 server1 kernel: [27446]   505 27446    61276     1381   0       0             0 php
Nov 15 03:17:23 server1 kernel: [27481]   505 27481    51564     1267   1       0             0 php
Nov 15 03:17:23 server1 kernel: [27530]   504 27530    56009     1467   0       0             0 php
Nov 15 03:17:23 server1 kernel: [27552]   505 27552    51215      966   0       0             0 php
Nov 15 03:17:23 server1 kernel: [27578]   505 27578    55113     1982   0       0             0 php
Nov 15 03:17:23 server1 kernel: [27592]   505 27592    55113     1403   0       0             0 php
Nov 15 03:17:23 server1 kernel: [27596]   505 27596    54984     1274   1       0             0 php
Nov 15 03:17:23 server1 kernel: [27599]   505 27599    55113     1879   1       0             0 php
Nov 15 03:17:23 server1 kernel: [27601]   505 27601    54073      954   0       0             0 php
Nov 15 03:17:23 server1 kernel: [27605]   505 27605    53995     1152   1       0             0 php
Nov 15 03:17:23 server1 kernel: [27608]   505 27608    54078     1264   0       0             0 php
Nov 15 03:17:23 server1 kernel: [27611]   505 27611    54984     1128   0       0             0 php
Nov 15 03:17:23 server1 kernel: [27615]    99 27615   334516     1232   0       0             0 httpd
Nov 15 03:17:23 server1 kernel: [27619]   505 27619    54073     1192   0       0             0 php
Nov 15 03:17:23 server1 kernel: [27620]    99 27620   334516     1173   0       0             0 httpd
Nov 15 03:17:23 server1 kernel: [27622]   505 27622    51215     1298   0       0             0 php
Nov 15 03:17:23 server1 kernel: [27623]    99 27623   334448     1139   1       0             0 httpd
Nov 15 03:17:23 server1 kernel: [27664]   505 27664    54944     1418   0       0             0 php
Nov 15 03:17:23 server1 kernel: [27665]   505 27665    55113     1320   0       0             0 php
Nov 15 03:17:23 server1 kernel: [27666]   505 27666    53995     1305   1       0             0 php
Nov 15 03:17:23 server1 kernel: [27668]   505 27668    53944     1156   0       0             0 php
Nov 15 03:17:23 server1 kernel: [27669]   505 27669    53338      495   0       0             0 php
Nov 15 03:17:23 server1 kernel: [27690]   505 27690    53962     1327   0       0             0 php
Nov 15 03:17:23 server1 kernel: [27692]   505 27692    53369      619   1       0             0 php
Nov 15 03:17:23 server1 kernel: [27718]   505 27718    54177     1357   1       0             0 php
Nov 15 03:17:23 server1 kernel: [27723]   505 27723    54278     1112   1       0             0 php
Nov 15 03:17:23 server1 kernel: [27728]   505 27728    53435      578   0       0             0 php
Nov 15 03:17:23 server1 kernel: [27731]   505 27731    55146     1602   1       0             0 php
Nov 15 03:17:23 server1 kernel: [27733]     0 27733    35555       32   0       0             0 crond
Nov 15 03:17:23 server1 kernel: [27736]   505 27736    53944     1107   0       0             0 php
Nov 15 03:17:23 server1 kernel: [27737]   505 27737    55113     2210   1       0             0 php
Nov 15 03:17:23 server1 kernel: [27738]   505 27738    53338      634   0       0             0 php
Nov 15 03:17:23 server1 kernel: [27741]   505 27741    53369      696   0       0             0 php
Nov 15 03:17:23 server1 kernel: [27744]   505 27744    53286      530   0       0             0 php
Nov 15 03:17:23 server1 kernel: [27746]   504 27746    53732     1166   1       0             0 php
Nov 15 03:17:23 server1 kernel: [27747]   505 27747    53338      562   1       0             0 php
Nov 15 03:17:23 server1 kernel: [27748]   505 27748    54550     1168   0       0             0 php
Nov 15 03:17:23 server1 kernel: [27749]   505 27749    54924     1798   0       0             0 php
Nov 15 03:17:23 server1 kernel: [27750]   505 27750    54651     1678   0       0             0 php
Nov 15 03:17:23 server1 kernel: [27751]   505 27751    33004      277   1       0             0 php
Nov 15 03:17:23 server1 kernel: [27752]   505 27752    54073      958   0       0             0 php
Nov 15 03:17:23 server1 kernel: [27753]   505 27753    54073     1032   0       0             0 php
Nov 15 03:17:23 server1 kernel: [27754]   505 27754    55113     1728   0       0             0 php
Nov 15 03:17:23 server1 kernel: [27756]   505 27756    54073     1329   1       0             0 php
Nov 15 03:17:23 server1 kernel: [27759]   505 27759    33570      313   0       0             0 php
Nov 15 03:17:23 server1 kernel: [27760]   505 27760    53402      609   0       0             0 php
Nov 15 03:17:23 server1 kernel: [27761]   505 27761    53765      934   1       0             0 php
Nov 15 03:17:23 server1 kernel: [27762]   505 27762    54078     1029   0       0             0 php
Nov 15 03:17:23 server1 kernel: [27763]   505 27763    54073      980   0       0             0 php
Nov 15 03:17:23 server1 kernel: [27767]   505 27767    54272     1121   1       0             0 php
Nov 15 03:17:23 server1 kernel: [27768]   505 27768    53369      676   1       0             0 php
Nov 15 03:17:23 server1 kernel: [27795]   505 27795    33768      545   0       0             0 php
Nov 15 03:17:23 server1 kernel: [27796]   505 27796    54353     1128   1       0             0 php
Nov 15 03:17:23 server1 kernel: [27797]   505 27797    54145      908   0       0             0 php
Nov 15 03:17:23 server1 kernel: [27798]   505 27798    54414      996   0       0             0 php
Nov 15 03:17:23 server1 kernel: [27799]   505 27799    53468      712   1       0             0 php
Nov 15 03:17:23 server1 kernel: [27802]   505 27802    53338      611   0       0             0 php
Nov 15 03:17:23 server1 kernel: [27809]   505 27809    54956     1484   0       0             0 php
Nov 15 03:17:23 server1 kernel: [27811]   505 27811    36362      461   0       0             0 php
Nov 15 03:17:23 server1 kernel: [27812]     0 27812    35555       24   0       0             0 crond
Nov 15 03:17:23 server1 kernel: [27814]   505 27814    53369      677   0       0             0 php
Nov 15 03:17:23 server1 kernel: [27815]   505 27815    54074     1114   1       0             0 php
Nov 15 03:17:23 server1 kernel: [27820]   505 27820    54651     1579   0       0             0 php
Nov 15 03:17:23 server1 kernel: [27821]   505 27821    54073     1058   1       0             0 php
Nov 15 03:17:23 server1 kernel: [27822]   505 27822    53896     1006   1       0             0 php
Nov 15 03:17:23 server1 kernel: [27824]   504 27824    33768      482   0       0             0 php
Nov 15 03:17:23 server1 kernel: [27855]   505 27855    53435      707   0       0             0 php
Nov 15 03:17:23 server1 kernel: [27856]   505 27856    53929     1052   0       0             0 php
Nov 15 03:17:23 server1 kernel: [27858]   505 27858    54944     1722   1       0             0 php
Nov 15 03:17:23 server1 kernel: [27859]   505 27859    55113     2026   1       0             0 php
Nov 15 03:17:23 server1 kernel: [27860]   505 27860    55113     1712   0       0             0 php
Nov 15 03:17:23 server1 kernel: [27861]   505 27861    33735      468   1       0             0 php
Nov 15 03:17:23 server1 kernel: [27862]   505 27862    54984     1669   0       0             0 php
Nov 15 03:17:23 server1 kernel: [27863]   505 27863    55220     1601   0       0             0 php
Nov 15 03:17:23 server1 kernel: [27865]   505 27865    54073      984   0       0             0 php
Nov 15 03:17:23 server1 kernel: [27867]   505 27867    54953     1773   0       0             0 php
Nov 15 03:17:23 server1 kernel: [27870]     0 27870    11802      482   1       0             0 update_quota_ca
Nov 15 03:17:23 server1 kernel: [27872]   505 27872    53402      725   0       0             0 php
Nov 15 03:17:23 server1 kernel: [27873]   505 27873    54274     1169   1       0             0 php
Nov 15 03:17:23 server1 kernel: [27875]   505 27875    54073      921   1       0             0 php
Nov 15 03:17:23 server1 kernel: [27879]   505 27879    54352     1332   0       0             0 php
Nov 15 03:17:23 server1 kernel: [27881]   505 27881    36362      554   0       0             0 php
Nov 15 03:17:23 server1 kernel: [27882]   505 27882    53863      980   1       0             0 php
Nov 15 03:17:23 server1 kernel: [27883]   505 27883    36362      500   1       0             0 php
Nov 15 03:17:23 server1 kernel: [27885]   505 27885    54719     1410   1       0             0 php
Nov 15 03:17:23 server1 kernel: [27886]   505 27886    53338      677   1       0             0 php
Nov 15 03:17:23 server1 kernel: [27887]   505 27887    54851     1289   1       0             0 php
Nov 15 03:17:23 server1 kernel: [27892]   505 27892    53765      881   1       0             0 php
Nov 15 03:17:23 server1 kernel: [27893]   505 27893    53338      527   1       0             0 php
Nov 15 03:17:23 server1 kernel: [27895]   505 27895    53534      750   1       0             0 php
Nov 15 03:17:23 server1 kernel: [27896]     0 27896    24521       97   0       0             0 cpsrvd-ssl
Nov 15 03:17:23 server1 kernel: [27897]   505 27897    54486     1439   1       0             0 php
Nov 15 03:17:23 server1 kernel: [27898]   505 27898    54073     1170   0       0             0 php
Nov 15 03:17:23 server1 kernel: [27899]   504 27899    54205     1136   0       0             0 php
Nov 15 03:17:23 server1 kernel: [27900]   504 27900    53600      825   1       0             0 php
Nov 15 03:17:23 server1 kernel: [27901]   505 27901    55113     1798   0       0             0 php
Nov 15 03:17:23 server1 kernel: [27902]   505 27902    54486     1304   0       0             0 php
Nov 15 03:17:23 server1 kernel: [27903]   505 27903    54550     1317   1       0             0 php
Nov 15 03:17:23 server1 kernel: [27904]   505 27904    53534      737   1       0             0 php
Nov 15 03:17:23 server1 kernel: [27905]   505 27905    55079     1952   1       0             0 php
Nov 15 03:17:23 server1 kernel: [27906]   504 27906    36362      502   0       0             0 php
Nov 15 03:17:23 server1 kernel: [27907]   505 27907    55050     1589   0       0             0 php
Nov 15 03:17:23 server1 kernel: [27909]   505 27909    53338      662   1       0             0 php
Nov 15 03:17:23 server1 kernel: [27914]   505 27914    54074     1187   1       0             0 php
Nov 15 03:17:23 server1 kernel: [27916]   505 27916    53369      592   1       0             0 php
Nov 15 03:17:23 server1 kernel: [27918]     0 27918    35555       26   1       0             0 crond
Nov 15 03:17:23 server1 kernel: [27919]   505 27919    54075     1225   1       0             0 php
Nov 15 03:17:23 server1 kernel: [27920]   505 27920    51215     1989   0       0             0 php
Nov 15 03:17:23 server1 kernel: [27923]     0 27923     2307        1   1       0             0 sh
Nov 15 03:17:23 server1 kernel: [27927]   505 27927    53369      623   1       0             0 php
Nov 15 03:17:23 server1 kernel: [27928]   505 27928    54073     1012   0       0             0 php
Nov 15 03:17:23 server1 kernel: [27929]   505 27929    53369      666   1       0             0 php
Nov 15 03:17:23 server1 kernel: [27930]   505 27930    54216     1070   1       0             0 php
Nov 15 03:17:23 server1 kernel: [27931]   505 27931    54073     1208   1       0             0 php
Nov 15 03:17:23 server1 kernel: [27933]   505 27933    28823      131   1       0             0 php
Nov 15 03:17:23 server1 kernel: [27934]   505 27934    54489     1647   1       0             0 php
Nov 15 03:17:23 server1 kernel: [27935]   505 27935    54272     1309   1       0             0 php
Nov 15 03:17:23 server1 kernel: [27936]   505 27936    53369      564   0       0             0 php
Nov 15 03:17:23 server1 kernel: [27937]   505 27937    55152     2162   0       0             0 php
Nov 15 03:17:23 server1 kernel: [27938]   505 27938    54140     1091   0       0             0 php
Nov 15 03:17:23 server1 kernel: [27940]   505 27940    53338      656   0       0             0 php
Nov 15 03:17:23 server1 kernel: [27941]   505 27941    52746      590   1       0             0 php
Nov 15 03:17:23 server1 kernel: [27942]   505 27942    53600      924   0       0             0 php
Nov 15 03:17:23 server1 kernel: [27943]   505 27943    32478      433   1       0             0 php
Nov 15 03:17:23 server1 kernel: [27944]   504 27944    54476     1388   0       0             0 php
Nov 15 03:17:23 server1 kernel: [27945]   505 27945    54078      944   1       0             0 php
Nov 15 03:17:23 server1 kernel: [27946]     0 27946     2307        2   1       0             0 sh
Nov 15 03:17:23 server1 kernel: [27947]   505 27947    54145      835   1       0             0 php
Nov 15 03:17:23 server1 kernel: [27948]   505 27948    54074     1310   0       0             0 php
Nov 15 03:17:23 server1 kernel: [27949]   505 27949    54073      983   1       0             0 php
Nov 15 03:17:23 server1 kernel: [27950]   505 27950    54785     1356   1       0             0 php
Nov 15 03:17:23 server1 kernel: [27951]   505 27951    53369      614   0       0             0 php
Nov 15 03:17:23 server1 kernel: [27952]   505 27952    53369      580   0       0             0 php
Nov 15 03:17:23 server1 kernel: [27953]   505 27953    53402      559   1       0             0 php
Nov 15 03:17:23 server1 kernel: [27954]   505 27954    54550     1606   1       0             0 php
Nov 15 03:17:23 server1 kernel: [27955]   505 27955    53369      657   1       0             0 php
Nov 15 03:17:23 server1 kernel: [27956]   505 27956    54138     1195   0       0             0 php
Nov 15 03:17:23 server1 kernel: [27957]   505 27957    31879      418   0       0             0 php
Nov 15 03:17:23 server1 kernel: [27958]   505 27958    28907      240   1       0             0 php
Nov 15 03:17:23 server1 kernel: [27962]   505 27962    54958     1785   0       0             0 php
Nov 15 03:17:23 server1 kernel: [27963]   505 27963    55113     1897   1       0             0 php
Nov 15 03:17:23 server1 kernel: [27964]   505 27964    53863     1083   0       0             0 php
Nov 15 03:17:23 server1 kernel: [27965]   505 27965    53369      588   0       0             0 php
Nov 15 03:17:23 server1 kernel: [27966]   505 27966    53468      732   0       0             0 php
Nov 15 03:17:23 server1 kernel: [27967]   505 27967    55151     2239   1       0             0 php
Nov 15 03:17:23 server1 kernel: [27968]   504 27968    28823      131   1       0             0 php
Nov 15 03:17:23 server1 kernel: [27971]   505 27971    53369      630   0       0             0 php
Nov 15 03:17:23 server1 kernel: [27974]   505 27974    54890     1480   0       0             0 php
Nov 15 03:17:23 server1 kernel: [27975]     0 27975    12683      650   0       0             0 mysqluserstore
Nov 15 03:17:23 server1 kernel: [27976]     0 27976     6668      222   1       0             0 dnsqueue
Nov 15 03:17:23 server1 kernel: [27977]   505 27977    53370      665   1       0             0 php
Nov 15 03:17:23 server1 kernel: [27979]   505 27979    28822       82   1       0             0 php
Nov 15 03:17:23 server1 kernel: [27980]   505 27980    32478      441   0       0             0 php
Nov 15 03:17:23 server1 kernel: [27984]   505 27984    28822       86   1       0             0 php
Nov 15 03:17:23 server1 kernel: [27985]   505 27985    29631      397   0       0             0 php
Nov 15 03:17:23 server1 kernel: [27987]   505 27987    28842      159   0       0             0 php
Nov 15 03:17:23 server1 kernel: [27991]   505 27991    29118      383   1       0             0 php
Nov 15 03:17:23 server1 kernel: [27993]   505 27993    28842      164   1       0             0 php
Nov 15 03:17:23 server1 kernel: [27995]   505 27995    28907      197   1       0             0 php
Nov 15 03:17:23 server1 kernel: [27996]   505 27996    28823      112   0       0             0 php
Nov 15 03:17:23 server1 kernel: [27997]   505 27997    32478      431   0       0             0 php
Nov 15 03:17:23 server1 kernel: [27998]   505 27998    30692      443   1       0             0 php
Nov 15 03:17:23 server1 kernel: [27999]   505 27999    28842      190   1       0             0 php
Nov 15 03:17:23 server1 kernel: [28008]   505 28008    28823      112   0       0             0 php
Nov 15 03:17:23 server1 kernel: [28013]   505 28013    28823       91   0       0             0 php
Nov 15 03:17:23 server1 kernel: [28018]     0 28018    18258     2722   1       0             0 cpses_tool
Nov 15 03:17:23 server1 kernel: [28020]   505 28020    28907      208   1       0             0 php
Nov 15 03:17:23 server1 kernel: [28021]   505 28021    28823      119   1       0             0 php
Nov 15 03:17:23 server1 kernel: [28023]   505 28023    28907      218   0       0             0 php
Nov 15 03:17:23 server1 kernel: [28024]   505 28024    28823      112   1       0             0 php
Nov 15 03:17:23 server1 kernel: [28025]   505 28025    28819       84   1       0             0 php
Nov 15 03:17:23 server1 kernel: [28026]   505 28026    28842      189   0       0             0 php
Nov 15 03:17:23 server1 kernel: [28027]   505 28027    28823      128   0       0             0 php
Nov 15 03:17:23 server1 kernel: [28028]   505 28028    32478      430   0       0             0 php
Nov 15 03:17:23 server1 kernel: [28029]   505 28029    28823      157   1       0             0 php
Nov 15 03:17:23 server1 kernel: [28033]   505 28033    28918      243   1       0             0 php
Nov 15 03:17:23 server1 kernel: [28034]   505 28034    28822       81   0       0             0 php
Nov 15 03:17:23 server1 kernel: [28036]   505 28036    30692      433   0       0             0 php
Nov 15 03:17:23 server1 kernel: [28038]   505 28038    28823      151   1       0             0 php
Nov 15 03:17:23 server1 kernel: [28039]   505 28039    28823      126   0       0             0 php
Nov 15 03:17:23 server1 kernel: [28040]   505 28040    28842      182   1       0             0 php
Nov 15 03:17:23 server1 kernel: [28041]   505 28041    28822       89   1       0             0 php
Nov 15 03:17:23 server1 kernel: [28042]   505 28042    28842      188   1       0             0 php
Nov 15 03:17:23 server1 kernel: [28043]   505 28043    28823      141   1       0             0 php
Nov 15 03:17:23 server1 kernel: [28044]   505 28044    23785       59   0       0             0 php
Nov 15 03:17:23 server1 kernel: [28045]   505 28045    28823      127   1       0             0 php
Nov 15 03:17:23 server1 kernel: [28046]   505 28046    30692      428   1       0             0 php
Nov 15 03:17:23 server1 kernel: [28047]   505 28047    28907      247   0       0             0 php
Nov 15 03:17:23 server1 kernel: [28048]   505 28048    28823      124   0       0             0 php
Nov 15 03:17:23 server1 kernel: [28049]   505 28049    28822      100   0       0             0 php
Nov 15 03:17:23 server1 kernel: [28050]   505 28050    28822       94   1       0             0 php
Nov 15 03:17:23 server1 kernel: [28051]   505 28051    30692      412   0       0             0 php
Nov 15 03:17:23 server1 kernel: [28052]   505 28052    28823      110   1       0             0 php
Nov 15 03:17:23 server1 kernel: [28053]    99 28053   334448     1069   0       0             0 httpd
Nov 15 03:17:23 server1 kernel: [28054]   505 28054    30692      350   0       0             0 php
Nov 15 03:17:23 server1 kernel: [28055]    99 28055   334516     1755   1       0             0 httpd
Nov 15 03:17:23 server1 kernel: [28059]   505 28059    28823      111   0       0             0 php
Nov 15 03:17:23 server1 kernel: [28060]   505 28060    28823      109   1       0             0 php
Nov 15 03:17:23 server1 kernel: [28061]   505 28061    28823      142   1       0             0 php
Nov 15 03:17:23 server1 kernel: [28062]   505 28062    29631      370   0       0             0 php
Nov 15 03:17:23 server1 kernel: [28063]   505 28063    28823      148   1       0             0 php
Nov 15 03:17:23 server1 kernel: [28064]   505 28064    28822       98   0       0             0 php
Nov 15 03:17:23 server1 kernel: [28065]   505 28065    28823      110   0       0             0 php
Nov 15 03:17:23 server1 kernel: [28066]   505 28066    30692      319   1       0             0 php
Nov 15 03:17:23 server1 kernel: [28069]   505 28069    28823      142   0       0             0 php
Nov 15 03:17:23 server1 kernel: [28070]   505 28070    28842      197   1       0             0 php
Nov 15 03:17:23 server1 kernel: [28071]   505 28071    28823      123   0       0             0 php
Nov 15 03:17:23 server1 kernel: [28072]   505 28072    28823      115   0       0             0 php
Nov 15 03:17:23 server1 kernel: [28073]   505 28073    28823      135   1       0             0 php
Nov 15 03:17:23 server1 kernel: [28074]   505 28074    26718       62   0       0             0 php
Nov 15 03:17:23 server1 kernel: [28112]   505 28112    28822       95   0       0             0 php
Nov 15 03:17:23 server1 kernel: [28114]   504 28114    28823      123   0       0             0 php
Nov 15 03:17:23 server1 kernel: [28115]   505 28115    29631      393   0       0             0 php
Nov 15 03:17:23 server1 kernel: [28119]   505 28119    28823      113   0       0             0 php
Nov 15 03:17:23 server1 kernel: [28120]     0 28120     2835       41   1       0             0 mysqld_safe
Nov 15 03:17:23 server1 kernel: [28131]   505 28131    28907      225   1       0             0 php
Nov 15 03:17:23 server1 kernel: [28132]   505 28132    28823      132   1       0             0 php
Nov 15 03:17:23 server1 kernel: [28133]   505 28133    28823      138   0       0             0 php
Nov 15 03:17:23 server1 kernel: [28134]   505 28134    28822       96   0       0             0 php
Nov 15 03:17:23 server1 kernel: [28135]   505 28135    28820      105   1       0             0 php
Nov 15 03:17:23 server1 kernel: [28136]   505 28136    29631      393   0       0             0 php
Nov 15 03:17:23 server1 kernel: [28137]   505 28137    28823      113   0       0             0 php
Nov 15 03:17:23 server1 kernel: [28139]   505 28139    28819      102   0       0             0 php
Nov 15 03:17:23 server1 kernel: [28141]     0 28141     2307       28   0       0             0 sh
Nov 15 03:17:23 server1 kernel: [28143]   505 28143    28820      106   1       0             0 php
Nov 15 03:17:23 server1 kernel: [28144]   505 28144    28842      193   1       0             0 php
Nov 15 03:17:23 server1 kernel: [28145]   505 28145    26718       63   0       0             0 php
Nov 15 03:17:23 server1 kernel: [28146]   505 28146    28823      109   1       0             0 php
Nov 15 03:17:23 server1 kernel: [28147]   505 28147    27757       69   0       0             0 php
Nov 15 03:17:23 server1 kernel: [28148]   505 28148    26718       60   1       0             0 php
Nov 15 03:17:23 server1 kernel: [28149]   505 28149    30692      385   0       0             0 php
Nov 15 03:17:23 server1 kernel: [28152]   505 28152    28823      115   0       0             0 php
Nov 15 03:17:23 server1 kernel: [28153]   505 28153    28819       86   1       0             0 php
Nov 15 03:17:23 server1 kernel: [28154]   505 28154    28823      153   1       0             0 php
Nov 15 03:17:23 server1 kernel: [28155]   505 28155    30155      434   0       0             0 php
Nov 15 03:17:23 server1 kernel: [28169]   505 28169    28823      111   0       0             0 php
Nov 15 03:17:23 server1 kernel: [28170]   505 28170    28823      122   0       0             0 php

From this messages log what do you think it could be? Some searches on google with similar messages suggest a swap issue?

heusdens · Nov 15, 2013

Hi

an update, the server just shut down again, it appears to be happening around 3am and 1pm.

Load averages shot up to : 95.77 67.52 31.48

Perhaps its the cron jobs? I look at the cpanel cron job configuration and crikey moses, it's set to run at 1 and 3.30 which is pretty much in line with when the server crashes.

This only started happening when WHM was upgraded to WHM 11.40.0 (build 19).

Any suggestions on how this can be fixed without disabling it?

kernow · Nov 15, 2013

Your server is crashing because its run out of memory ( OOM). Check WHM >>> daily process log and confirm what's eating up your RAM

cPanelMichael · Nov 15, 2013

Hello

Please review the following thread the next time your server load spikes:

Troubleshooting High Load on Linux Systems

It should help you to diagnose the cause of the high load.

Thank you.

heusdens · Nov 15, 2013

Well what I've done is adjust the cpbackup config to run once a week. Therefore my assumption is that tomorrow morning at 3.30am my server won't crash.

If my assumption is correct and my normal scheduled backups don't crash my server then why would cpbackup be crashing the server?

How would one diagnose that. What logs would one check in?

cPanelMichael · Nov 15, 2013

You could manually generate backups via:

Code:

/usr/local/cpanel/bin/backup --force

Then, monitor the load and resource usage as backups generate to determine if that's the cause of the problem.

Thank you.

sahostking · Nov 17, 2013

I have the same issue since 11.40, Going to try and schedule backups at a different time as it crashes at the same time each day.
Will provide feedback in a 2 days.

heusdens · Nov 17, 2013

I've been monitoring since my last post, what I did was disable backups, and will run them manually every other day.

Since then everything has been running smooth. However this is the weekend so the true test to see whether it was the cron jobs is yet to be determined as traffic will begin to pick up again during the week.

heusdens · Nov 17, 2013

Hi cPanelMichael

I have an update, I did a force backup via /usr/local/cpanel/bin/backup --force and monitored server load using top to see if it was indeed contributing.

The interesting thing I noticed was that the pigz process was consuming 100 to even 200% of cpu shooting the server average load to >2.

I assume this is not normal and probably the reason why the server crashed.

Do you have any advice?

cPanelMichael · Nov 19, 2013

A server load higher than 2 is not necessarily considered a high load. Was this slowing the server to the point where it was noticeable or failing to respond? Was the disk I/O (sar -u command) higher than normal?

Thank you.

Thread starter	Similar threads	Forum	Replies	Date
W	Server wont reboot. Stuck on some login screen. (Almalinux 8)	Operating Systems	10	Sep 8, 2023
O	Is it safe to restart server during MySQL upgrade to v8.0?	Operating Systems	1	Jul 18, 2023
W	OVH Server stuck at Grub prompt when booted to HDD	Operating Systems	2	Jul 14, 2023
P	cPanel Server keeps kernel panicking	Operating Systems	0	Jun 8, 2011
X	cpsrvd keeps failing and server keeps restarting.	Operating Systems	17	Aug 16, 2006

Search

Search

Server keeps crashing in last 3 days - Swap issue?

heusdens

Active Member

heusdens

Active Member

kernow

Well-Known Member

cPanelMichael

Administrator

heusdens

Active Member

cPanelMichael

Administrator

sahostking

Well-Known Member

heusdens

Active Member

heusdens

Active Member

cPanelMichael

Administrator