sendip -p ipv4 -is 192.168.1.81 -p udp -us 5070 -ud 5060 -d "Hello" -v 192.168.1.81

This should send a “Hello” message over udp to port 5060 using an ipv4 address.

Here the result:

Added 26 options
Initializing module ipv4
Initializing module udp
Finalizing module udp
Finalizing module ipv4
Final packet data:
45 00 00 21   E..!
34 E0 00 00   4...
FF 11 02 F9   ....
C0 A8 01 51   ...Q
C0 A8 01 51   ...Q
13 CE 13 C4   ....
00 0D 30 7D   ..0}
48 65 6C 6C   Hell
6F   o
Sent 33 bytes to 192.168.1.81
Freeing module ipv4
Freeing module udp

It is pretty awesome that you can build about any protocol packet for quick testing using just bash …

Find the presentations below:

– Elastix World 2011, Mexico D.F. Nov 3-4 – “Negociación de codecs en Asterisk”

– FSL, Vallarta, México. Nov 5 – “FreeSWITCH – Asterisk con esteroides”

– 4K Conference, Buenos Aires, Argentina, Nov 24  – “Manejo de medios en FreeSWITCH“

What is a core dump?

Sometimes problems with FreeSWITCH, Asterisk or just about any other program in Linux are hard to debug by just looking at the logs. Sometimes the process crashes and you don’t have a chance to use the CLI to look at stats. The logs may not reveal anything particularly interesting, or just some minimal information. Sometimes you need to dig deeper into the state of the program to poke around and see what is going on inside. Linux process core dumps are meant for that.

The most typical case for a core dump is when a process dies violently and unexpectedly. For example, if a programmer does something like this:

*(int *)0 = 0;

Is usually not that straight forward, it may be the that the programmer did not expect certain variable to contain a NULL (0) value. The process is killed by the Linux kernel because by default, a Linux process does not map the memory address 0 to anything. This causes a page fault in the processor which is trapped by the kernel, the kernel then sees that the given process does not have anything mapped at address 0 and then sends the SIGSEGV (Unix signal) to the process. This is called a segmentation fault.

The default signal handler for SIGSEGV dumps the memory of the process and kills the process. This memory dump contains all the memory for this process (and all the threads belonging to that process) and that is what is used to determine what went wrong with the program that attempted to reference an invalid address (0 in this case, but any invalid address can cause it).

You can learn more about it here: http://en.wikipedia.org/wiki/Segmentation_fault

How can I make sure the core dump will be saved?

Each process has a limit for how big this core can be. If the limit is exceeded no core dump will be saved. By default this limit is 0!, that means no core will be dumped by default.

Before starting the process you must use “ulimit“. The “ulimit” command sets various limits for the current process. If you execute it from the bash shell, that means the limits are applied to your bash shell process. This also means any processes that you start from bash will inherit those limits (because they are child processes from your bash shell).

ulimit -a

That shows you all the limits for your bash shell. In order to guarantee that a core will be dumped you must set the “core file size” limit to “unlimited”.

ulimit -c unlimited

If you are starting the process from an init script or something like that, the init script has to do it. Some programs are smart enough to raise their limits themselves, but is always better to make sure you have unlimited core file size for your bash shell. You may then want to add those ulimit instructions inside your $HOME/.bashrc file.

Where is the core dump saved?

Each process has a “working directory”. See http://en.wikipedia.org/wiki/Working_directory

That is where the process core dump will be saved by default. However, some system-wide settings affect where the core is dumped.

“/proc/sys/kernel/core_pattern” and “/proc/sys/kernel/core_uses_pid” are 2 files that control the base file name pattern for the core, and whether the core name will be appended with the PID (process ID).

The recommended settings are:

mkdir -p /var/core
echo "/var/core/core" > /proc/sys/kernel/core_pattern
echo 1 > /proc/sys/kernel/core_uses_pid

You can confirm what you just did with:

cat /proc/sys/kernel/core_pattern

cat /proc/sys/kernel/core_uses_pid

This settings will cause any process in the system that crashes, to dump the core at:

/var/core/core.

What if I just want a core dump without killing the process?

In some situations, if a process becomes unresponsive or the response times are not ideal. For example, you try to execute CLI commands in Asterisk or FreeSWITCH but there is no output, or worst, your command line prompt gets stuck. The process is still there, it may be even processing calls, but some things are taking lot of time or just don’t get done. You can use “gdb” (The GNU debugger) to dump a core of the process without killing the process and almost with no disruption of the service. I say almost because for a large process, dumping a core may take a second or two, in that time the process is freezed by the kernel, so active calls may drop some audio (if you’re debugging some real time audio system like Asterisk or FreeSWITCH).

The trick to do it fast is to first create a file with the GDB commands required to dump the core.

Latest versions of CentOS include (with the gdb RPM package) the “gcore” command to do everything for you. You only need to execute:

gcore $(pidof freeswitch)

To dump the core of the running process.

If you are in a system that does not include gcore, you can do the following:

echo -ne "generate-core-file\ndetach\nquit" > gdb-instructions.txt

The 3 instructions added to the file are:

generate-core-file
detach
quit

This will do exactly what we want. Generate the core file for the attached process, then detach from the process (to let it continue) and then quit gdb.

You then use GDB (you may need to install it with “yum install gdb”) to attach to the running process, dump the core, and get out as fast as possible.

gdb /usr/local/freeswitch/bin/freeswitch $(pidof freeswitch) -x gdb-instructions.txt

The arguments to attach to the process include the original binary that was used to start it and the PID of the running process. The -x switch tells GDB to execute the commands in the file after attaching.

The core will be named core.<pid> by default and the path is not affected by the /proc/sys/kernel settings of the system.

This core can now be used by the developer to troubleshoot the problem.

Sometimes though, the developer will be more interested in a full back trace (stack trace), because the core dump itself can’t be easily examined in any other box than the one where was generated, therefore it might be up to you to provide that stack trace, which you can do with:

gdb /usr/local/freeswitch/bin/freeswitch core.
(gdb) set logging file my_back_trace.txt
(gdb) thread apply all bt full
(gdb) quit

Then send the file my_back_trace.txt to the developer, or analyze it yourself, sometimes is easy to spot the problems even without development experience!

It is sometimes required to analyze protocol messages in a given link in order to troubleshoot call problems. Wanpipemon makes possible to create a pcap file with all D-channel data which can then be examined using Wireshark.

What makes it so convenient is that this method is independent from the application you’re using (Asterisk, FreeSWITCH or whatever). You basically have the capability to tap into the D-channel, pretty much like using tcpdump to monitor TCP/IP traffic.

wanpipemon -i w1g1 -pcap -pcap_file isdn.pcap -prot ISDN -full -systime -c trd

You can refer to the Sangoma Wiki for other details (and SS7 tracing commands too): http://wiki.sangoma.com/wanpipe-wireshark-pcap-pri-bri-wan-t1-e1-tracing

Next time your D-channel link does not come up, fire up wanpipemon and take a look in Wireshark to what’s going on!

I was clearly able to switch to the correct directory with cd /path/to/repo && git pull, but it kept failing.

It seems git hooks run with some git environment variables like GIT_DIR that affect the new instance of git being spawned. The easiest solution I found was to specify –git-dir when doing the pull:

cd /path/to/repo
git --git-dir=/path/to/repo/.git pull

And finally I could move on after being stuck like half an hour

– To disable SELinux. Edit /etc/selinux/config and change the SELINUX policy from enforcing to

SELINUX=disabled

Be aware that you are disabling security “features”, whatever that means. So you may want to read this other article about disabling SELinux.

I wasn’t lucky enough to have SELinux disabled as an option. Developing with SELinux enabled is a good idea so you can notice compatibility problems with SELinux in your development environment before a customer that really *needs* SELinux enabled discovers them in your software, which, from the customer point of view is a plain bug.

The first problem I noticed after some CentOS upgrade, change the hard-drive or something along those lines, was that the command “ping” wasn’t working, it’s been a while since I had the problem so I don’t quite remember the exact error when pinging other systems, but it was most likely something like permission denied. Probably other network commands did not work either, but I could just notice ping at that moment. So, I used the good old strace to find out what was causing the problem.

The underlying ping problem was because the open() system call was failing with EACCESS when trying to open /etc/hosts. However I was able to “cat /etc/hosts”. So it wasn’t a simple permission problem, but a bit more complex SELinux problem. Eventually I found out that the solution was:

restorecon reset /etc/hosts

At which point the SELinux security context for this file got screwed up? I certainly don’t know. But that command restored it. The Z option of the ls command will show you the SELinux security context for any file.

ls -Z /etc/hosts

The second problem was that some libraries of one of the programs I am responsible for were not being loaded. Again, the problem was due to permission denied errors, this time when loading the shared libraries that required text relocation.

The solution was to recompile the shared libraries with the -fPIC option.

I am sure SELinux has its uses, however I have the feeling that sometimes makes things more complicated than needed in some environments. I recommend reading this blog post and particularly the comments in there.

At least on recent libc implementations in Linux, the mutex object seems to have a member named “__owner”. Let me show you what I recently saw when debugging a deadlocked application.

(gdb) f 4
#4  0x0805ab46 in ACE_OS::mutex_lock (m=0xa074248) at include/ace/OS.i:1406
1406      ACE_OSCALL_RETURN (ACE_ADAPT_RETVAL (pthread_mutex_lock (m), ace_result_),
(gdb) p *m
$8 = {__data = {__lock = 2, __count = 0, __owner = 17828, __kind = 0, __nusers = 1, {__spins = 0, __list = {__next = 0x0}}},
  __size = "\002\000\000\000\000\000\000\000�E\000\000\000\000\000\000\001\000\000\000\000\000\000", __align = 2}

We can see that the __owner is 17828. This number is the LWP (Light-weight process) id of the thread holding the lock. Now you can go to examine that thread stack and find out why that thread is also stuck.

This example also brings up a regular point of confusion for some Linux application developers. What is the difference between LWP and POSIX thread id ( the pthread_t type in pthread.h)?. The difference is that pthread_t is a user space concept, is simply an identifier for the thread library implementing POSIX threads to refer to the thread and its resources, state etc. However the LWP is an implementation detail of how the Linux kernel implements threads, which is done through the “thread group” concept and LWP’s, that are processes that share memory pages and other resources with the other processes in the same thread group.

From the Linux kernel point of view the pthread_t value doesn’t mean anything, the LWP id is how you identify threads in the kernel, and they share the same numbering as regular processes, since LWPs are just a special type of process. Knowing this is useful when using utilities like strace. When you want to trace a particular thread of a multi threaded application, you need to provide the LWP of the thread you want to trace, a common mistake is to provide the process id, which in a multithreaded application the process id is just the LWP of the first thread in the application (the one that started executing main()).

Here is how you get each identifier in a C program:

#include 
#include 
#include 

int main()
{
  pthread_t tid = pthread_self();
  int sid = syscall(SYS_gettid);
  printf("LWP id is %d\n", sid);
  printf("POSIX thread id is %d\n", tid);
  return 0;
}

It’s important to note that getting the POSIX thread id is much faster than the LWP, because pthread_self() is just a library call and libc most likely has this value cached somewhere in user space, no need to go down to the kernel. As you can see, getting the LWP requires a call to the syscall() function, which effectively executes the requested system call, this is expensive (well, compared with the time required to enter a simple user space function).

Even when the application is simple in nature, learning about V.110 to get it to work was interesting

Today I made the project public ( thanks to Tzury Bar Yochay from Regulus Labs) in google code:

http://code.google.com/p/sbridge/

Hopefully somebody else will find it useful.

However, when you ship non-open source software or you’re just concerned with the size of all the debugging information in lots of libraries, you want to separate the debugging information from the binary holding the program/library itself. In Windows this is the default behavior you get with the well known PDB (Program Data Base) files. For Linux though, you need some tricks to get the debugging information separate. This is of course what most distributions do, they include an extra package with debugging information, so when you install a package you get just the binary code, then, if you need to debug it you download the debugging package.

If you ever need this, you can follow the instructions in this web page to get it to work:

http://sources.redhat.com/gdb/current/onlinedocs/gdb_17.html#SEC166

The way I solved it for our internal build system is just to always compile with -ggdb3 and then:

1. Create a copy of the debugging symbols in a separate binary

objcopy --only-keep-debug somelibrary.so somelibrary.so.dbg 

2. Remove the debugging information from the code binary.

objcopy --strip-debug somelibrary.so

3. Add a reference to the code binary so gdb knows where to look for the debugging information

objcopy --add-gnu-debuglink somelibrary.so.dbg somelibrary.so

This last step is simply putting a file name reference inside the ELF binary so GDB (or some other debugger) knows which file name will have the debugging information for this .so (or an executable if that’s what you’re building). In the red hat web page more advanced techniques are explained to make sure you don’t end up with a version mismatch between the debugging information and your library or executable.

#!/bin/bash
autoheader
libtoolize --force --copy
aclocal
automake -f --copy --add-missing
autoconf

The –force –copy for libtoolize and -f –copy –add-missing for automake will help you to not depend on symbolic links that are created by libtoolize and automake and that may not be present on the target machine where your code will be built.

I suppose there is a valid reason to not use those options, but for me, that saved me a lot of hassle.