I don’t know 🙂
But people has asked me this a couple of times lately and my answer is always “I don’t know”. However ps can give you more information about it. In fact, this works for any application you have and you want to debug why is going crazy.
First, check which thread (Asterisk is a multi threaded application) is going crazy.
# ps -LlFm -p `pidof asterisk`
That should show you the % of CPU being used by each Asterisk thread in the column named “C”, then write down the LWP colum value for the thread you are interested on. (LWP is a light weight process number, roughly speaking, the thread id). Now that you have the thread id, you need to know what that thread is doing.
# pstack `pidof asterisk` > /tmp/asterisk.stack.txt
That will cause the asterisk process to dump the stack state to the /tmp/asterisk.stack.txt file. If you don’t have the pstack command google for it, I think in CentOS is as easy as yum install pstack.
Then open the file and search for the LWP that you just wrote down. Hopefully you will find some hints that let you know how to avoid it or at least a lot more information to post in bugs.digium.com
UPDATE:
One of the guys who asked this question later told me what he found:
Thread 10 (Thread 0x41d8f940 (LWP 3406)):
#0 0x00000033ce2ca436 in poll () from /lib64/libc.so.6
#1 0x00000000004933c0 in ast_io_wait ()
#2 0x00002aaabd9510cd in network_thread ()
#3 0x00000000004f8b2c in dummy_start ()
#4 0x00000033cee06367 in start_thread () from /lib64/libpthread.so.0
#5 0x00000033ce2d2f7d in clone () from /lib64/libc.so.6
A quick grep -rI “network_thread” in the Asterisk source code reveals this function belongs to chan_iax.c, disabling chan_iax.so in modules.conf is a good workaround to his problem, however further debugging would be needed to determine why the monitor thread is looping like that.
can someone let me know whey this is taking more and more Core, sometimes shooting upto 500% t0 600% for asterisk process…
Thread 11 (Thread 0x2b26ca678940 (LWP 26060)):
#0 0x00000031c48cc116 in poll () from /lib64/libc.so.6
#1 0x0000000000437aa7 in ast_waitfor_nandfds ()
#2 0x0000000000437e42 in ast_waitfor ()
#3 0x000000000044f1a2 in ?? ()
#4 0x000000000044f596 in ast_waitstream ()
#5 0x0000000000474281 in ?? ()
#6 0x000000000046dce5 in pbx_exec ()
#7 0x00000000004717e7 in ?? ()
#8 0x0000000000471c73 in ast_spawn_extension ()
#9 0x0000000000475c17 in ?? ()
#10 0x0000000000477019 in ?? ()
#11 0x00000000004a12de in ?? ()
#12 0x00000031c500683d in start_thread () from /lib64/libpthread.so.0
#13 0x00000031c48d503d in clone () from /lib64/libc.so.6
Hi there
This helped me a lot, in my case the thread using CPU was asterisk itself and had to do with an selinux issue. Fixed now 🙂 Many thanks.
Just a side note to those who don’t find pstack, in CentOS7(2016) pstack is part of gdb, install with `yum install gdb`.
Please let me know how you have fixed this issue.
I have no idea what I’m looking at here, but I suddenly have the 100% CPU problem. Problem started today but restoring from a backup from 3 days ago didn’t resolve it. The output of ps -LlFm -p `pidof asterisk` was:
4 – asterisk 2481 2480 – 90 17 – – – 17335 – 51616 – 14:53 pts/0 00:00:51 /usr/
1 D asterisk – – 2498 90 17 49 – – – log_wa – 0 14:53 – 00:00:51 –
So I assume the LWP I’m looking for is 2498? If so, this is the result of pstack:
Thread 5 (Thread -1212511312 (LWP 2498)):
#0 0x0050f7a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2
#1 0x005eb1d4 in poll () from /lib/tls/libc.so.6
#2 0x0070b677 in __libc_res_nsend () from /lib/libresolv.so.2
#3 0x0070a126 in __libc_res_nquery () from /lib/libresolv.so.2
#4 0x0070a3f8 in __libc_res_nquerydomain () from /lib/libresolv.so.2
#5 0x0070a95e in __libc_res_nsearch () from /lib/libresolv.so.2
#6 0x00a272ce in _nss_dns_gethostbyname3_r () from /lib/libnss_dns.so.2
#7 0x00a27538 in _nss_dns_gethostbyname_r () from /lib/libnss_dns.so.2
#8 0x0060a454 in gethostbyname_r@@GLIBC_2.1.2 () from /lib/tls/libc.so.6
#9 0x080b22c6 in ast_gethostbyname ()
What the heck does that tell me and what’s my next step? Thanks.
This helped me track down 100% CPU issue. Turned out it was streaming music-on-hold setup I did. I deleted that setting and it went back down to normal. — thanks for the post.
I dont see where the issue lies, am I missing something?
Thread 6 (Thread 0x7f6e673e1700 (LWP 24181)):
#0 0x00007f6f7defa343 in poll () from /lib64/libc.so.6
#1 0x000000000043f599 in ast_waitfor_nandfds ()
#2 0x000000000044656f in __ast_request_and_dial ()
#3 0x0000000000489cd4 in ast_pbx_outgoing_exten ()
#4 0x0000000000478ce4 in action_originate ()
#5 0x00000000004752ea in process_message ()
#6 0x0000000000476c9f in do_message ()
#7 0x0000000000476d68 in session_do ()
#8 0x00000000004b746a in dummy_start ()
#9 0x00007f6f7e8759d1 in start_thread () from /lib64/libpthread.so.0
#10 0x00007f6f7df03b6d in clone () from /lib64/libc.so.6
Thread 5 (Thread 0x7f6e62379700 (LWP 24182)):
#0 0x00007f6f7defa343 in poll () from /lib64/libc.so.6
#1 0x000000000043f599 in ast_waitfor_nandfds ()
#2 0x000000000043faa7 in ast_waitfor_n ()
#3 0x00007f6e6d2ec6cd in dial_exec_full () from /usr/lib/asterisk/modules/app_dial.so
#4 0x00007f6e6d2f1115 in dial_exec () from /usr/lib/asterisk/modules/app_dial.so
#5 0x0000000000487850 in pbx_extension_helper ()
#6 0x0000000000488b62 in __ast_pbx_run ()
#7 0x000000000048a2d9 in pbx_thread ()
#8 0x00000000004b746a in dummy_start ()
#9 0x00007f6f7e8759d1 in start_thread () from /lib64/libpthread.so.0
#10 0x00007f6f7df03b6d in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x7f6e66005700 (LWP 24197)):
#0 0x00007f6f7defa343 in poll () from /lib64/libc.so.6
#1 0x000000000043f599 in ast_waitfor_nandfds ()
#2 0x000000000044656f in __ast_request_and_dial ()
#3 0x0000000000489cd4 in ast_pbx_outgoing_exten ()
#4 0x0000000000478ce4 in action_originate ()
#5 0x00000000004752ea in process_message ()
#6 0x0000000000476c9f in do_message ()
#7 0x0000000000476d68 in session_do ()
#8 0x00000000004b746a in dummy_start ()
#9 0x00007f6f7e8759d1 in start_thread () from /lib64/libpthread.so.0
#10 0x00007f6f7df03b6d in clone () from /lib64/libc.so.6
Pingback: » Asterisk CPU usage at 100% with FreePBX – SysAdminMan Blog
I’m glad it helped you Mike. Indeed, using strace on the LWP is another cool way of seeing what your multi-threaded app is doing.
This really helps, in my case after find out the thread id consuming CPU, strace on the thread found there’s futex flood, and finally figure out the problem was caused by leap second on June 30, 2012 at 23:59:60.
Really appreciate.
Pingback: CPU 100% Астериск | Блог Мироничева Якова
Pingback: Asterisk FAQs » Blog Archive » Asterisk-cpu utilization > 60 %
It’s fixed – no CPU usage alerts thus far. For future reference this was in the h323.conf / ooh323.conf where I fixed the settings. After that did an asterisk -rx ‘restart now’ and issue has not recurred. I guess this is all caused because I ran asterisk make samples during install.
Good news 🙂
Phemnneoal breakdown of the topic, you should write for me too!
🙂 Thank you for the reply (and your useful blog post).
I fixed the module by actually configuring it to do nothing. By default there is a mypeer1 and myfriend1 configured which I commented out. I also added my servers IP.
For the last 10 hours it hasn’t been acting up, so if I don’t post again, it is fixed.
Now you either:
– disable the module
– fix the module
– pay someone to fix the module
So:
Thread 13 (Thread 0xb6e73b90 (LWP 1724)):
#0 0xf57fe416 in __kernel_vsyscall ()
#1 0x4e0969d1 in select () from /lib/libc.so.6
#2 0xb6eccbb9 in ooSocketSelect () from /usr/lib/asterisk/modules/chan_ooh323.$
#3 0xb6eb9a35 in ooH2250Receive () from /usr/lib/asterisk/modules/chan_ooh323.$
#4 0xb6eba418 in ooProcessFDSETsAndTimers () from /usr/lib/asterisk/modules/ch$
#5 0xb6eba74e in ooMonitorChannels () from /usr/lib/asterisk/modules/chan_ooh3$
#6 0xb6fb92a7 in ooh323c_stack_thread () from /usr/lib/asterisk/modules/chan_o$
#7 0x0810071b in dummy_start ()
#8 0x4e1d4832 in start_thread () from /lib/libpthread.so.0
#9 0x4e09de0e in clone () from /lib/libc.so.6
Now what?
One possible reason to spin like crazy in select()-based systems is having more than 1024 file descriptors in the process. (increasing limit with ulimit -n does not help and in fact may do things worst).
Much thanks!
Thread 13 (Thread 0xb6e74b90 (LWP 3128)):
#0 0xf57fe416 in __kernel_vsyscall ()
#1 0x4e0969d1 in select () from /lib/libc.so.6
#2 0xb6ecdbb9 in ooSocketSelect () from /usr/lib/asterisk/modules/chan_ooh323.so
#3 0xb6ebaa35 in ooH2250Receive () from /usr/lib/asterisk/modules/chan_ooh323.so
#4 0xb6ebb418 in ooProcessFDSETsAndTimers () from /usr/lib/asterisk/modules/chan_ooh323.so
#5 0xb6ebb74e in ooMonitorChannels () from /usr/lib/asterisk/modules/chan_ooh323.so
#6 0xb6fba2a7 in ooh323c_stack_thread () from /usr/lib/asterisk/modules/chan_ooh323.so
#7 0x0810071b in dummy_start ()
#8 0x4e1d4832 in start_thread () from /lib/libpthread.so.0
#9 0x4e09de0e in clone () from /lib/libc.so.6
Yet another reason to not use Ubuntu.
Ubuntu 9.10 does not have pstack. It’s not in the Karmic repository, so doing an “apt-get install pstack” failed to find the package.