环境:
SLES11SP3
3.0.101-0.47.52-default
现象:
系统操作卡顿,kswapd0 进程占用大量 CPU,top 查看 kswapd0 结果如下:
SLES11SP3-53:~ # top | grep -e COMMAND -e kswapd0
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
30 root 20 0 0 0 0 R 96 0.0 32976:47 kswapd0
30 root 20 0 0 0 0 R 100 0.0 32976:50 kswapd0
30 root 20 0 0 0 0 R 99 0.0 32976:53 kswapd0
30 root 20 0 0 0 0 R 100 0.0 32976:56 kswapd0
物理内存有空闲,但已经开始使用 swap
SLES11SP3-53:~ # free
total used free shared buffers cached
Mem: 16336624 10877416 5459208 0 2600 28116
-/+ buffers/cache: 10846700 5489924
Swap: 5236732 10744 5225988
SLES11SP3-53:~ # free
total used free shared buffers cached
Mem: 16336624 10876920 5459704 0 2600 28276
-/+ buffers/cache: 10846044 5490580
Swap: 5236732 10716 5226016
atop 看到 swout 数值过高标红
PAG | scan 329237 | steal 529 | stall 0 | swin 524 | swout 120 |
分析:
vm.min_free_kbytes 为5000000(来自配置文件 /etc/sysctl.conf),比其它环境大很多,16GB内存虚拟机默认安装值仅有 67584。
根据参考资料,当 free 小于 low 时会启动 kswapd 进行内存回收,因为 low (通过vm.min_free_kbytes 计算得到)设置过高,kswapd 经过回收后仍未达到要求 high 值,所以 kswapd 持续运行。
SLES11SP3-53:~ # cat /proc/zoneinfo | sed -n '/Normal/,${p}' | grep -A 7 "pages free"
pages free 1059529
min 1016538
low 1270672
high 1524807
scanned 0
spanned 3407872
present 3361280
nr_free_pages 1059529
SLES11SP3-53:~ # sysctl vm.min_free_kbytes
vm.min_free_kbytes = 5000000
解决:
vm.min_free_kbytes 设置为默认值67584,同时去掉 /etc/sysctl.conf 中的配置项。系统卡顿现象消失,kswapd0 不再占用 CPU。
SLES11SP3-53:~ # sysctl -w vm.min_free_kbytes=67584
vm.min_free_kbytes = 67584
SLES11SP3-53:~ # cat /proc/zoneinfo | sed -n '/Normal/,${p}' | grep -A 7 "pages free"
pages free 1057888
min 13740
low 17175
high 20610
scanned 0
spanned 3407872
present 3361280
nr_free_pages 1057888
SLES11SP3-53:~ # top | grep -e COMMAND -e kswapd0
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
30 root 20 0 0 0 0 S 0 0.0 32994:12 kswapd0
30 root 20 0 0 0 0 S 0 0.0 32994:12 kswapd0
30 root 20 0 0 0 0 S 0 0.0 32994:12 kswapd0
30 root 20 0 0 0 0 S 0 0.0 32994:12 kswapd0
参考:
https://www.suse.com/support/kb/doc/?id=7017605
http://kernel.taobao.org/index.php?title=Kernel_Documents/mm_sysctl
https://www.kernel.org/doc/Documentation/sysctl/vm.txt