Swap pages per second in/out

GoWilkes · Mar 24, 2014

I'm looking at the Munin report, under "Swap in/out - by day".

What is considered "good" when it comes to this section? I just migrated to a new server, and it's showing several times throughout the day that I'm either between 0 and -4.0, or between 0 and +3.0. I can't look at the stats on my old server (they have already aged out of Munin), but I don't remember it having ANY swap.

The CPU and RAM usage are well within normal limits. I have 4G of RAM, and "apps" usually stays between 1.6G and 3.0G. I also have two 2.4GHz CPUs, and the load usually stays between 0.4 and 0.5 (with a max today of 1.17 during peak hours).

With this hardware, should I be using swap at all? If not, what's causing it?

Sys Admin · Mar 26, 2014

Do you have any type of RAID on the new server? Please provide the full output of:

free -mt

cat /proc/cpuinfo

lscpu

sar -q

sar -r

iostat

GoWilkes · Mar 26, 2014

No, to my knowledge there's no RAID.

Here is the output you requested. I'm doing this at 3:30am, when the traffic is typically the lowest; do you need it during peak hours, instead?:

Code:

# free -mt
             total       used       free     shared    buffers     cached
Mem:          3881       3810         71          0        271       2206
-/+ buffers/cache:       1331       2549
Swap:         2047        121       1925
Total:        5928       3932       1996

Code:

# cat /proc/cpuinfo
processor       : 0
vendor_id       : GenuineIntel
cpu family      : 6
model           : 42
model name      : Intel(R) Xeon(R) CPU E31270 @ 3.40GHz
stepping        : 7
cpu MHz         : 3392.412
cache size      : 8192 KB
fpu             : yes
fpu_exception   : yes
cpuid level     : 13
wp              : yes
flags           : fpu de tsc msr pae cx8 sep cmov pat clflush mmx fxsr sse sse2 ss ht syscall nx lm rep_good aperfmperf unfair_spinlock pni pclmulqdq ssse3 cx16 pcid sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer hypervisor lahf_lm arat epb pln pts dts
bogomips        : 6784.82
clflush size    : 64
cache_alignment : 64
address sizes   : 36 bits physical, 48 bits virtual
power management:

processor       : 1
vendor_id       : GenuineIntel
cpu family      : 6
model           : 42
model name      : Intel(R) Xeon(R) CPU E31270 @ 3.40GHz
stepping        : 7
cpu MHz         : 3392.412
cache size      : 8192 KB
fpu             : yes
fpu_exception   : yes
cpuid level     : 13
wp              : yes
flags           : fpu de tsc msr pae cx8 sep cmov pat clflush mmx fxsr sse sse2 ss ht syscall nx lm rep_good aperfmperf unfair_spinlock pni pclmulqdq ssse3 cx16 pcid sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer hypervisor lahf_lm arat epb pln pts dts
bogomips        : 6784.82
clflush size    : 64
cache_alignment : 64
address sizes   : 36 bits physical, 48 bits virtual
power management:

Code:

# lscpu
Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                2
On-line CPU(s) list:   0,1
Thread(s) per core:    1
Core(s) per socket:    1
Socket(s):             2
NUMA node(s):          1
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 42
Stepping:              7
CPU MHz:               3392.412
BogoMIPS:              6784.82
Hypervisor vendor:     Xen
Virtualization type:   para
L1d cache:             32K
L1i cache:             32K
L2 cache:              256K
L3 cache:              8192K
NUMA node0 CPU(s):     0,1

Code:

# sar -q
Linux 2.6.32-431.5.1.el6.x86_64 (server1.example.com)    03/26/2014      _x86_64_(2 CPU)

12:00:01 AM   runq-sz  plist-sz   ldavg-1   ldavg-5  ldavg-15
12:10:01 AM         2       204      0.02      0.11      0.09
12:20:01 AM         2       197      0.21      0.14      0.10
12:30:01 AM         5       205      0.33      0.21      0.12
12:40:01 AM         2       198      0.07      0.15      0.13
12:50:01 AM         2       197      0.04      0.10      0.12
01:00:01 AM         3       218      0.07      0.10      0.10
01:10:01 AM         4       195      0.22      0.14      0.10
01:20:01 AM         2       196      0.15      0.16      0.12
01:30:01 AM         3       200      0.07      0.18      0.17
01:40:01 AM         2       212      0.19      0.21      0.18
01:50:01 AM         2       206      0.04      0.14      0.16
02:00:01 AM         3       202      0.12      0.13      0.15
02:10:01 AM         3       195      0.16      0.13      0.14
02:20:01 AM         2       198      0.08      0.14      0.15
02:30:01 AM         4       196      0.03      0.09      0.13
02:40:01 AM         2       196      0.03      0.04      0.08
02:50:01 AM         2       208      0.25      0.16      0.10
03:00:01 AM         3       190      0.12      0.14      0.09
03:10:01 AM         2       197      0.09      0.09      0.09
03:20:01 AM         2       188      0.25      0.15      0.10
03:30:01 AM         3       192      0.05      0.08      0.08
Average:            3       200      0.12      0.13      0.12

Code:

# sar -r
12:00:01 AM kbmemfree kbmemused  %memused kbbuffers  kbcached  kbcommit   %commi                                                                                                 t
12:10:01 AM    117856   3856504     97.03    178260   2292152   1864560     30.7                                                                                                 1
12:20:01 AM    128532   3845828     96.77    180780   2298528   1727312     28.4                                                                                                 5
12:30:01 AM    110456   3863904     97.22    182816   2308996   1772864     29.2                                                                                                 0
12:40:01 AM     62140   3912220     98.44    185252   2352588   1705716     28.1                                                                                                 0
12:50:01 AM     66372   3907988     98.33    186092   2356304   1680684     27.6                                                                                                 8
01:00:01 AM     34952   3939408     99.12    188968   2328928   2035512     33.5                                                                                                 3
01:10:01 AM    116472   3857888     97.07    182364   2312368   1638760     26.9                                                                                                 9
01:20:01 AM     52800   3921560     98.67    184788   2370712   1692752     27.8                                                                                                 8
01:30:01 AM     53616   3920744     98.65    186752   2364744   1708268     28.1                                                                                                 4
01:40:01 AM     97264   3877096     97.55    176848   2326128   1922004     31.6                                                                                                 6
01:50:01 AM     79656   3894704     98.00    179012   2331968   1794340     29.5                                                                                                 6
02:00:01 AM    108632   3865728     97.27    178092   2313560   1762932     29.0                                                                                                 4
02:10:01 AM    127436   3846924     96.79    180128   2318684   1682480     27.7                                                                                                 1
02:20:01 AM     93912   3880448     97.64    181788   2328084   1722140     28.3                                                                                                 7
02:30:01 AM     83464   3890896     97.90    183540   2336868   1688508     27.8                                                                                                 1
02:40:01 AM     93792   3880568     97.64    185412   2340540   1687460     27.8                                                                                                 0
02:50:01 AM     42456   3931904     98.93    184364   2334248   1932212     31.8                                                                                                 3
03:00:01 AM     95196   3879164     97.60    184612   2344428   1609236     26.5                                                                                                 1
03:10:01 AM     90688   3883672     97.72    186528   2354392   1692308     27.8                                                                                                 8
03:20:01 AM     57448   3916912     98.55    279416   2271124   1530716     25.2                                                                                                 1
03:30:01 AM     53464   3920896     98.65    278072   2260352   1618000     26.6                                                                                                 5
Average:        84124   3890236     97.88    192090   2325986   1736608     28.6                                                                                                 1

Code:

# iostat
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           6.75    0.63    4.41    1.42    0.08   86.71

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
xvdb              0.41         1.05         2.63     145852     366576
xvda             53.89       183.23       631.54   25501168   87895698

cPanelMichael · Mar 31, 2014

Hello

I do not see any performance issues based on the output from "iostat" that you provided.

Thank you.

GoWilkes · Apr 2, 2014

For whatever reason, at around 8:50am EST today my disk latency, disk utilization, RAM, and CPU usage skyrocketed. I usually have a CPU load of around 0.5, and it's currently 77!

I had made a few minor Exim changes through the WHM Basic Configuration editor, so I thought that might have been the culprit. I restored, and even turned off Exim entirely, with no change, so I guess that wasn't it.

You can see exactly what I mean here:

Code:

# sar -q
Linux 2.6.32-431.5.1.el6.x86_64 (server1.example.com)    04/02/2014      _x86_64_(2 CPU)

12:00:01 AM   runq-sz  plist-sz   ldavg-1   ldavg-5  ldavg-15
12:10:01 AM         1       200      0.06      0.17      0.17
12:20:01 AM         2       195      0.12      0.21      0.18
12:30:01 AM         4       195      0.06      0.17      0.17
12:40:01 AM         2       198      0.27      0.19      0.18
12:50:01 AM         2       206      0.21      0.19      0.18
01:00:01 AM         4       196      0.11      0.12      0.15
01:10:01 AM         2       194      0.01      0.13      0.16
01:20:01 AM         2       190      0.06      0.15      0.16
01:30:01 AM         3       189      0.06      0.14      0.16
01:40:01 AM         2       199      0.02      0.08      0.13
01:50:01 AM         2       197      0.25      0.13      0.10
02:00:01 AM         2       196      0.03      0.06      0.08
02:10:01 AM         3       202      0.15      0.12      0.09
02:20:01 AM         2       185      0.09      0.05      0.06
02:30:01 AM         4       191      0.18      0.08      0.04
02:40:01 AM         2       190      0.00      0.01      0.01
02:50:01 AM         2       191      0.03      0.03      0.00
03:00:01 AM         2       184      0.07      0.11      0.03
03:10:01 AM         2       187      0.19      0.16      0.10
03:20:01 AM         2       187      0.06      0.10      0.09
03:30:01 AM         3       183      0.02      0.08      0.08
03:40:02 AM         2       200      0.04      0.06      0.07

03:40:02 AM   runq-sz  plist-sz   ldavg-1   ldavg-5  ldavg-15
03:50:01 AM         2       195      0.08      0.09      0.08
04:00:01 AM         3       193      0.07      0.20      0.17
04:10:01 AM         2       190      0.05      0.09      0.12
04:20:01 AM         2       200      0.42      0.32      0.20
04:30:01 AM         3       205      0.18      0.20      0.18
04:40:01 AM         2       192      0.16      0.21      0.18
04:50:01 AM         2       198      0.22      0.18      0.16
05:00:01 AM         2       192      0.16      0.16      0.15
05:10:01 AM         2       184      0.17      0.11      0.10
05:20:01 AM         2       186      0.18      0.11      0.09
05:30:01 AM         3       195      0.07      0.06      0.06
05:40:01 AM         2       194      0.12      0.14      0.11
05:50:01 AM         2       206      0.19      0.13      0.10
06:00:01 AM         3       201      0.04      0.12      0.10
06:10:01 AM         2       201      0.11      0.08      0.08
06:20:01 AM         4       213      0.06      0.12      0.10
06:30:01 AM         3       210      0.13      0.19      0.13
06:40:01 AM         2       206      0.24      0.22      0.17
06:50:01 AM         2       205      0.05      0.14      0.16
07:00:01 AM         2       210      0.17      0.19      0.18
07:10:01 AM         3       207      0.10      0.17      0.17
07:20:01 AM         2       228      0.41      0.27      0.20

07:20:01 AM   runq-sz  plist-sz   ldavg-1   ldavg-5  ldavg-15
07:30:01 AM         2       212      0.23      0.18      0.17
07:40:01 AM         4       213      0.03      0.18      0.18
07:50:01 AM         2       231      0.18      0.24      0.20
08:00:01 AM         4       246      0.09      0.27      0.25
08:10:01 AM         2       231      0.48      0.28      0.22
08:20:01 AM         1       299      0.54      0.68      0.50
08:30:01 AM         5       241      0.80      1.16      0.90
08:40:01 AM         2       286      0.52      0.95      0.96
08:50:01 AM         2       255     10.68      7.60      3.85
09:00:01 AM         1       347     15.27      8.13      5.00
09:10:01 AM         2       224      5.09      5.79      5.44
09:20:01 AM         2       313     33.26     15.00      8.44
09:30:01 AM         2       289     13.55     25.06     19.79
09:40:01 AM         2       295     25.29     31.82     27.66
09:50:01 AM         1       312     37.19     28.17     26.51
10:00:01 AM         2       232      4.27     11.82     19.89
10:10:01 AM         2       267     11.29      8.02     13.50
10:20:01 AM         2       288     28.31     35.00     25.58
10:30:02 AM         3       222      2.95      9.96     17.47
10:40:02 AM         3       235      4.73      4.73     11.03
10:50:01 AM         3       215      1.26      4.04      8.30
11:00:01 AM         0       220      5.29      7.98      8.47

11:00:01 AM   runq-sz  plist-sz   ldavg-1   ldavg-5  ldavg-15
11:10:01 AM         2       231      4.63      4.48      6.49
11:20:01 AM         2       309     73.26     54.97     27.27
11:30:01 AM         2       271      1.68     11.20     17.52
11:40:01 AM         3       244      5.08      5.15     11.27
11:50:01 AM         2       313     53.20     29.30     18.71
12:00:01 PM         2       223      1.43     10.97     16.00
12:10:01 PM         3       254      3.21      3.42      9.33
12:20:01 PM         2       232      3.04      2.55      6.05
12:30:01 PM         4       214      4.32      3.39      4.64
12:40:13 PM         5       313     27.70     24.75     14.27
12:50:01 PM         2       288     10.56     15.85     16.26
01:00:01 PM         3       216      1.05      3.70      9.50
01:10:01 PM         1       241      1.92      5.92      8.35
01:20:01 PM         2       316     54.19     34.36     18.99
01:30:01 PM         2       213      3.33     10.49     14.00
01:40:01 PM         3       234      4.86      4.99      9.14
01:50:01 PM         2       243      9.73     11.92     11.20
02:00:01 PM         2       220      5.78      8.41      9.21
02:10:01 PM         2       231      3.87      4.82      6.95
02:20:01 PM         2       224      4.76      4.61      5.62
02:30:01 PM         3       228      1.41      3.70      5.37
02:40:01 PM         3       252      3.10      4.01      4.60

02:40:01 PM   runq-sz  plist-sz   ldavg-1   ldavg-5  ldavg-15
02:50:01 PM         2       231      3.07      3.79      4.53
03:00:03 PM         2       319     27.38     14.55      8.95
03:10:01 PM         2       226      4.17      8.28      9.53
03:20:01 PM         2       358     16.61     16.15     12.25
03:30:01 PM         2       293     30.77     39.20     26.62
03:40:01 PM         2       238      3.64     16.06     21.69
03:50:01 PM         2       249     11.61     24.09     23.27
04:00:03 PM         1       349     40.80     33.00     25.51
04:10:01 PM         2       289     24.49     26.66     24.34
Average:            2       231      6.72      6.78      6.36

Iostat doesn't look much different than when I first posted, though:

Code:

# iostat
Linux 2.6.32-431.5.1.el6.x86_64 (server1.example.com)    04/02/2014      _x86_64_(2 CPU)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           5.89    0.63    3.92    2.66    0.07   86.84

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
xvdb              0.27         1.24         1.02     976420     805592
xvda             48.77       169.81       557.79  134086558  440438298

In Munin, all of the graphs are super high. VMStat shows "I/O Sleep" at 9.52, where it's usually at 0. CPU Usage - iowait is at 149.2, and it's usually at 0.5.

IO Service Time also had a major shift on the chart, but I don't know how to read or explain it; in Munin, the red line is usually stable halfway between 1e03 and 1e04, but now it is erratic and halfway between 1e06 and 1e07.

Disk Utilization in Munin is usually at around 500m, but it's currently at 95.54 (not 95m, but 95; so, 200 times higher than usual).

Disk Latency is usually around 180m, but is currently at 2.49.

If it matters, I'm using WHM 11.42.0 (build 23).

Any suggestions? Any changes to cPanel at 8:50am that could cause this?

GoWilkes · Apr 2, 2014

We think that this problem was hardware failure, not software, so this is all probably irrelevant now :-(

Man, this has been a bad day...

cPanelMichael · Apr 7, 2014

GoWilkes said:
We think that this problem was hardware failure, not software, so this is all probably irrelevant now :-(

Thank you for updating us with the outcome.

Thread starter	Similar threads	Forum	Replies	Date
N	Safe to turn off Swap space with 256GB of RAM with 50-70GB of RAM used at all times?	Operating Systems	2	May 27, 2023
R	Swap memory full	Operating Systems	10	Jan 6, 2023
E	Increase swap disk disk on CentOS 7 from 2 GB to 4 GB - how to update /etc/fstab	Operating Systems	3	Jan 10, 2021
M	Swap's partition size in CloudLinux	Operating Systems	5	May 15, 2020
	Server Load 4.17 and swap 100%	Operating Systems	2	Jan 24, 2020

Search

Search

Swap pages per second in/out

GoWilkes

Well-Known Member

Sys Admin

Well-Known Member

GoWilkes

Well-Known Member

cPanelMichael

Administrator

GoWilkes

Well-Known Member

GoWilkes

Well-Known Member

cPanelMichael

Administrator