cloudstack下libvirtd服务无响应问题

2019-10-13 21:14:50王振洲
[root@node5 log]# top -p 6485
top -p 6485
top - 09:19:41 up 12 days, 22:27, 1 user, load average: 3.05, 5.07, 6.64
Tasks:  1 total,  0 running,  1 sleeping,  0 stopped,  0 zombie
Cpu(s): 4.8%us, 1.4%sy, 0.0%ni, 93.1%id, 0.6%wa, 0.0%hi, 0.1%si, 0.0%st
Mem: 264420148k total, 182040780k used, 82379368k free,  834232k buffers
Swap: 8388600k total,    92k used, 8388508k free, 100453708k cached

  PID USER   PR NI VIRT RES SHR S %CPU %MEM  TIME+ COMMAND
 6485 root   20  0 984m 12m 4440 R 100.2 0.0 844:22.68 libvirtd       #cpu占用100%,无法释放,影响系统稳定性

杀进程

[root@node5 log]# kill -9 6485
[root@node5 log]# kill -9 6485
[root@master log]# ps ax |grep libvirtd  #检查进程依然存在
 6485 ?    R  863:37 libvirtd --daemon -l
[root@node5 ~]# libvirtd -d -l --config /etc/libvirt/libvirtd.conf
libvirtd:错误:Unable to initialize network sockets。查看 /var/log/messages 或者运行不带 --daemon 的命令查看更多信息。
[root@node5 ~]# netstat -antp |grep 16509
tcp    0   0 0.0.0.0:16509        0.0.0.0:*          LISTEN   3658/libvirtd    
tcp    1   0 192.168.14.25:16509     192.168.14.22:8717     CLOSE_WAIT -          
tcp    1   0 192.168.14.25:16509     192.168.14.20:5152     CLOSE_WAIT -          
tcp    1   0 192.168.14.25:16509     192.168.14.10:39359     CLOSE_WAIT -          
tcp    0   0 :::16509          :::*            LISTEN   3658/libvirtd    
tcp    39   0 ::1:16509          ::1:19715          CLOSE_WAIT - 

经过上述操作,初步判断libvirtd陷入了hang死状态。

追踪进程

[root@node5 log]#strace -f libvirtd
[pid 107570] close(23058)        = -1 EBADF (Bad file descriptor)
[pid 107570] close(23059)        = -1 EBADF (Bad file descriptor)
[pid 107570] close(23060)        = -1 EBADF (Bad file descriptor)
[pid 107570] close(23061)        = -1 EBADF (Bad file descriptor)
[pid 107570] close(23062)        = -1 EBADF (Bad file descriptor)
[pid 107570] close(23063)        = -1 EBADF (Bad file descriptor)
[pid 107570] close(23064)        = -1 EBADF (Bad file descriptor)
[pid 107570] close(23065)        = -1 EBADF (Bad file descriptor)
[pid 107570] close(23066)        = -1 EBADF (Bad file descriptor)
[pid 107570] close(23067)        = -1 EBADF (Bad file descriptor)
[pid 107570] close(23068)        = -1 EBADF (Bad file descriptor)
[pid 107570] close(23069)        = -1 EBADF (Bad file descriptor)
[pid 107570] close(23070)        = -1 EBADF (Bad file descriptor)
[pid 107570] close(23071)        = -1 EBADF (Bad file descriptor)
^C[pid 107570] close(23072 <unfinished ...>
Process 107559 detached
Process 107560 detached
Process 107561 detached
Process 107562 detached
Process 107563 detached
Process 107564 detached
Process 107565 detached
Process 107566 detached
Process 107567 detached
Process 107568 detached
Process 107569 detached
Process 107570 detached

父进程6485在不断的产生和关闭子进程,并返回错误信息。Bad file descriptor的原因(如何触发的,谁触发的)? 循环为何无法退出?问题如何再现?