Core

The FreeSWITCH core is contained in the src/*.c files, everything that you see there is considered “core”. The FreeSWITCH core provides common services and protocols that are re-used among many different call control or media protocols. For example, the core provides several useful API abstractions:

• OS abstraction based on top of the APR project (Apache Portable Runtime). There is a thin layer of abstraction (src/switch_apr.c) that prevents introducing a direct dependency between the rest of the system and the APR library. This means the rest of the system *DOES NOT* use APR directly, they use the API exposed in src/switch_apr.c for operating system abstraction primitives to create sockets, threads, manage memory etc.
• RTP stack (src/switch_rtp.c) and an API to create rtp streams, read/write to them etc. This provides access to RTP as RTP is used by many other signaling protocols such as SIP, H.323, Megaco etc, so it makes sense to have it in the core.
• Call switching (src/switch_ivr.c, src/switch_ivr_originate.c) API routines to create a new call (session) towards a destination without explicitly knowing the low-level protocol details. A module can therefore use switch_ivr_originate() API to place a call in SIP, H.323, SS7, PRI etc, using the same consistent API provided by the core.
• Media playing API (src/switch_ivr_play_say.c, src/switch_ivr_say.c) that allows you to play a file, insert tones etc etc on a given session (regardless of the signaling protocol)
• Session and channel management (src/switch_core_session.c, src/switch_channel.c) that allows to perform call control operations (answer a channel, hangup a channel) and low-level I/O operations (write a frame of media, read a frame of media) on sessions (regardless of the underlying signaling and media protocols).

(Image taken from the FreeSWITCH wiki long ago, can’t recall the URL anymore)

Sessions and Channels

The core abstraction for a call leg is the switch_core_session_t opaque structure. This contains all the necessary information for a given call leg. The session contains a channel. The channel is simply a lower-level representation of the call leg that contains information about the underlying protocols and data structures used to interface with that call leg (ie, contains function pointers to functions that allow the call to be answered, hangup, provide media etc) and the state of the session.

Sessions in FreeSWITCH live in their own thread. A typical call involves 2 sessions (the inbound session and then an outbound session, both of them “bridged”). Each session lives in its own thread, looping through a finite state machine (the session channel states go from CS_INIT, last state is CS_DESTROY, there are other states such as CS_ROUTING, CS_EXCHANGE_MEDIA etc etc. These states determine what the session is doing, each of the state handlers to perform the operation (ie, hangup) is executed in the session thread. It is possible however that other threads want to send a message to indicate something, or read/write information about a session in it. In order to do this in a safe way, there are several APIs that can be used to get a locked pointer to a session that allows you to peek into the session data safely without fearing the session thread will terminate and destroy the session while you’re looking at it or doing something with it from another thread, the main core function for this is:

switch_core_session_t *switch_core_session_locate(const char *uuid);

Every session has a UUID (universal identifier) that is unique across computers and across time (at least for practical purposes). You can use that UUID to ask the core to find a session in the system with that UUID. The core will attempt to find the session (in an internal hash table) and if it’s found, it will lock it and return it to you. You can then use the session to retrieve data, set data or perform operations on it or in its underlying switch_channel_t (which is obtained through switch_core_session_get_channel()) without fear of the session thread to destroy the session (for example if someone sets the state to CS_DESTROY). You must remember to unlock the session when done, and you should do that fast enough (not more than a few milliseconds) otherwise you’re blocking other threads from accessing that session if they need to do something with it. Unlock the session with:

switch_core_session_rwunlock();

This function unlocks the session and this releases it so other threads can use it (including destroy it).

Be aware that there are other core functions that return a session locked, such as switch_core_session_get_partner(). It is your responsibility to verify if the function you are using returns a locked session and then use switch_core_session_rwunlock() to unlock it when done.
Modules

FreeSWITCH is highly modular. The FreeSWITCH engine/core source code is contained in the src/ directory. The plugins or modules source code are contained in the src/module/ directory (the only exception is mod_freetdm, which is contained at libs/freetdm/mod_freetdm, there are some legacy reasons why this was done this way, but there is talks in the FreeSWITCH open source community about moving it to the right place in src/mod/endpoints/ folder).

There are different types of modules/plugins (in the future, I’ll refer to them simply as modules). There are endpoints, formats, loggers, event handlers, dialplans, TTS engines etc etc. Pretty much every directory under src/mod/ is a type of module. Each type of module registers an “interface” with the FreeSWITCH core. Great care has been taken to make proper abstractions and do not expose “module-specific” data into the FreeSWITCH core (src/*.c) files. The modules make use of FreeSWITCH core API primitives to request core services, the FreeSWITCH core uses the abstract interface exposed by different type of modules for performing operations. For example, the module mod_sofia is an endpoint module. Therefore mod_sofia code is contained at src/mod/endpoints/mod_sofia/

FreeTDM

The FreeTDM library is a modular API that allows applications to initialize different types of TDM/Analog signaling stacks, place/receive calls and read/write media. It supports signaling modules for SS7, ISDN(PRI/BRI), Analog, MFCR2 etc. It also supports a I/O modules to support multiple different hardware manufacturers. The 2 most prominent I/O modules are src/ftmod/ftdm_wanpipe/ftmod_wanpipe.c (For Sangoma Wanpipe cards) and the src/ftmod/ftmod_zt/ftmod_zt.c (For DAHDI-enabled cards such as Sangoma and Digium).

The FreeTDM project is part of the same source tree as FreeSWITCH, however FreeTDM does not depend on FreeSWITCH. FreeSWITCH also does not depend on FreeTDM, the glue that links them together is mod_freetdm, which is an endpoint module for FreeSWITCH that allows FreeSWITCH to place calls in SS7, PRI, MFC-R2 and Analog telephony networks. Note that the mod_freetdm module is just a plugin/extension to FreeSWITCH and it is a “user” of the freetdm library. As such, you should never expose internal opaque details of freetdm to mod_freetdm or worst, to FreeSWITCH. The freetdm.h header is the main public header that is used by freetdm API users. There are other headers under a private/ folder in freetdm that are not meant to be used by the API users, only for internal use of other freetdm C files/components.

Memory Allocation

When allocating memory within FreeSWITCH you must ask yourself where to allocate from. FreeSWITCH uses APR memory pools (wrapped on the switch_memory_pool_t) data structure. You can request a completely new memory pool for yosuelf, but that may be a waste, instead ask yourself whether the memory you’re allocating will be associated to a particular session (call leg) and whether needs to persist. For example, memory for a data structure to keep track of RTCP statistics of an RTP stream, will always be associated to the RTP stream, which in turn is associated to the session. In such cases it is recommended to use the session memory pool, which is automatically destroyed at the end of the call.

In the other hand, if the memory you are allocating must persist beyond the call life cycle, you will be better off requesting a new memory pool and then allocating from it, but now you’re responsible to destroy the pool when you’re done with your object life cycle. Remember it is not possible to free memory allocated from a pool until you destroy the whole pool completely.

Some functions to remember (from src/include/switch_core.h and src/switch_core_memory.c):

switch_core_session_get_pool() – Returns the memory pool associated with a given session
switch_core_session_alloc() – Allocate x amount of bytes from a given session memory pool
switch_core_session_sprintf() – Allocates a new string with the provided format using the session memory pool
switch_core_session_strdup() – Duplicates a new string using the session memory pool
switch_core_new_memory_pool() – Creates a new pool ready to start allocating objects
switch_core_destroy_memory_pool() – Destroy the given pool, you better be sure no one will be needing any of the objects allocated from that pool
switch_core_alloc() – Allocate memory from a given memory pool
switch_safe_free() – Free memory allocated directly using malloc()/calloc(), testing for NULL pointer
For all the session-related allocation functions, you must be sure the allocated object will NOT outlive the session.

Remember all switch allocation functions already initialize memory, you do not need to call memset() after allocation

There are a few cases where it’s justifiable to use standard memory allocation functions such as strdup()

strdup() is ok when you need a quick duplicate of another string but you have no session pointer or it’s something called often and short-lived, and therefore you do not want to use the session pointer with switch_core_session_strdup() because that would leave the memory allocated until the end of the call

String Manipulation

switch_copy_string() – Safely copy a string, use this instead of strcpy() or strncpy(). It will guarantee the output is null-terminated.
switch_set_tring() – Use this to initialize a string buffer of a fixed length. DO NOT USE THIS with char* pointers as the implementation uses sizeof() and sizeof(char*) is only 4 or 8!!
switch_toupper() – Convert a string to uppercase
switch_strstr() – Find a string in a substr
switch_safe_atoi() – Turn a string into a number, testing for NULL and defaulting to a given value in case of NULL
More gems can be found in src/include/switch_utils.h, please check there before hand-coding your own function or check if you can use one from switch_utils.h instead of using direct libc functions!!

Booleans

switch_true() – Use this to test if a string is true (ie, “true”, “enabled”, “on”, etc, all are considered true), this is useful when parsing configuration files
switch_false() – Use this to test if a string is false (ie “false”, “disabled” etc), this is useful when parsing configuration files
SWITCH_TRUE / SWITCH_FALSE – Use this along with switch_bool_t instead of using integers

ClueCon 2015

I just got back from attending ClueCon 2015 in Chicago. This year the FreeSWITCH team presented their video conferencing support in version 1.6. Pretty neat stuff!

This year my presentation was about “Scaling FreeSWITCH Performance”. One of the highlights is that using an alternative memory allocator may (or may not) improve drastically the performance of your FreeSWITCH instance.

As every year, I attended in August 2012  and August 2013 to this telephony developer conference  in Chicago (organized by the FreeSWITCH developers).

In 2012 I presented about using FreeSWITCH as an SBC (Session Border Controller)

Find the presentation here: http://www.moythreads.com/congresos/cluecon2012/

You can also view it in SlideShare: http://www.slideshare.net/MoisesSilva6/cluecon-2012kickasssbc

In 2013 I presented about SIP Testing

Find the presentation here: http://www.moythreads.com/congresos/cluecon2013/

You can also view it in SlideShare: http://www.slideshare.net/MoisesSilva6/sip-testing-with-freeswi

Having said that, still showed some stability issues and it became a burden to maintain because it is a patch to Asterisk.

Something you may find interesting is that a bit later after I wrote the tapping feature for Asterisk, I also did it for FreeSWITCH

It has been more stable in FreeSWITCH, mostly due to the fact that FreeSWITCH TDM abstraction is modular and it has been much more easy to maintain a tapping module rather than a patch. You can find the module in FreeSWITCH’s tree at libs/freetdm/src/ftmod/ftmod_pritap/ftmod_pritap.c

In addition to the TDM tapping module, I also wrote an RTP tapping module called mod_oreka that can be used for tapping any media stream on FreeSWITCH (SIP, TDM, H.323 etc)

See press release from OrecX

Find the presentations below:

– Elastix World 2011, Mexico D.F. Nov 3-4 – “Negociación de codecs en Asterisk”

– FSL, Vallarta, México. Nov 5 – “FreeSWITCH – Asterisk con esteroides”

– 4K Conference, Buenos Aires, Argentina, Nov 24  – “Manejo de medios en FreeSWITCH“

What is a core dump?

Sometimes problems with FreeSWITCH, Asterisk or just about any other program in Linux are hard to debug by just looking at the logs. Sometimes the process crashes and you don’t have a chance to use the CLI to look at stats. The logs may not reveal anything particularly interesting, or just some minimal information. Sometimes you need to dig deeper into the state of the program to poke around and see what is going on inside. Linux process core dumps are meant for that.

The most typical case for a core dump is when a process dies violently and unexpectedly. For example, if a programmer does something like this:

*(int *)0 = 0;

Is usually not that straight forward, it may be the that the programmer did not expect certain variable to contain a NULL (0) value. The process is killed by the Linux kernel because by default, a Linux process does not map the memory address 0 to anything. This causes a page fault in the processor which is trapped by the kernel, the kernel then sees that the given process does not have anything mapped at address 0 and then sends the SIGSEGV (Unix signal) to the process. This is called a segmentation fault.

The default signal handler for SIGSEGV dumps the memory of the process and kills the process. This memory dump contains all the memory for this process (and all the threads belonging to that process) and that is what is used to determine what went wrong with the program that attempted to reference an invalid address (0 in this case, but any invalid address can cause it).

You can learn more about it here: http://en.wikipedia.org/wiki/Segmentation_fault

How can I make sure the core dump will be saved?

Each process has a limit for how big this core can be. If the limit is exceeded no core dump will be saved. By default this limit is 0!, that means no core will be dumped by default.

Before starting the process you must use “ulimit“. The “ulimit” command sets various limits for the current process. If you execute it from the bash shell, that means the limits are applied to your bash shell process. This also means any processes that you start from bash will inherit those limits (because they are child processes from your bash shell).

ulimit -a

That shows you all the limits for your bash shell. In order to guarantee that a core will be dumped you must set the “core file size” limit to “unlimited”.

ulimit -c unlimited

If you are starting the process from an init script or something like that, the init script has to do it. Some programs are smart enough to raise their limits themselves, but is always better to make sure you have unlimited core file size for your bash shell. You may then want to add those ulimit instructions inside your $HOME/.bashrc file.

Where is the core dump saved?

Each process has a “working directory”. See http://en.wikipedia.org/wiki/Working_directory

That is where the process core dump will be saved by default. However, some system-wide settings affect where the core is dumped.

“/proc/sys/kernel/core_pattern” and “/proc/sys/kernel/core_uses_pid” are 2 files that control the base file name pattern for the core, and whether the core name will be appended with the PID (process ID).

The recommended settings are:

mkdir -p /var/core
echo "/var/core/core" > /proc/sys/kernel/core_pattern
echo 1 > /proc/sys/kernel/core_uses_pid

You can confirm what you just did with:

cat /proc/sys/kernel/core_pattern

cat /proc/sys/kernel/core_uses_pid

This settings will cause any process in the system that crashes, to dump the core at:

/var/core/core.

What if I just want a core dump without killing the process?

In some situations, if a process becomes unresponsive or the response times are not ideal. For example, you try to execute CLI commands in Asterisk or FreeSWITCH but there is no output, or worst, your command line prompt gets stuck. The process is still there, it may be even processing calls, but some things are taking lot of time or just don’t get done. You can use “gdb” (The GNU debugger) to dump a core of the process without killing the process and almost with no disruption of the service. I say almost because for a large process, dumping a core may take a second or two, in that time the process is freezed by the kernel, so active calls may drop some audio (if you’re debugging some real time audio system like Asterisk or FreeSWITCH).

The trick to do it fast is to first create a file with the GDB commands required to dump the core.

Latest versions of CentOS include (with the gdb RPM package) the “gcore” command to do everything for you. You only need to execute:

gcore $(pidof freeswitch)

To dump the core of the running process.

If you are in a system that does not include gcore, you can do the following:

echo -ne "generate-core-file\ndetach\nquit" > gdb-instructions.txt

The 3 instructions added to the file are:

generate-core-file
detach
quit

This will do exactly what we want. Generate the core file for the attached process, then detach from the process (to let it continue) and then quit gdb.

You then use GDB (you may need to install it with “yum install gdb”) to attach to the running process, dump the core, and get out as fast as possible.

gdb /usr/local/freeswitch/bin/freeswitch $(pidof freeswitch) -x gdb-instructions.txt

The arguments to attach to the process include the original binary that was used to start it and the PID of the running process. The -x switch tells GDB to execute the commands in the file after attaching.

The core will be named core.<pid> by default and the path is not affected by the /proc/sys/kernel settings of the system.

This core can now be used by the developer to troubleshoot the problem.

Sometimes though, the developer will be more interested in a full back trace (stack trace), because the core dump itself can’t be easily examined in any other box than the one where was generated, therefore it might be up to you to provide that stack trace, which you can do with:

gdb /usr/local/freeswitch/bin/freeswitch core.
(gdb) set logging file my_back_trace.txt
(gdb) thread apply all bt full
(gdb) quit

Then send the file my_back_trace.txt to the developer, or analyze it yourself, sometimes is easy to spot the problems even without development experience!

Anyways, the topic of my session this time was media handling in FreeSWITCH. We’ll be doing some work in in that area at Sangoma in the following months so I thought I’d kill two birds with one stone.

Presentation here:
PPT: http://www.moythreads.com/congresos/cluecon2011/media-handling-freeswitch-final.ppt

PDF: http://www.moythreads.com/congresos/cluecon2011/media-handling-freeswitch-final.pdf

http://svn.openzap.org/svn/openzap/trunk/

I also created some basic documentation about how to set it up: http://wiki.freeswitch.org/wiki/OpenZAP_OpenR2

This means that from now on FreeSWITCH will support MFC-R2 signaling with the same stack that Asterisk is using since 1.6.2

I still need to do some work on the documentation and lots of stress testing, but you can start playing with it and bugging me if it does not work

The power point presentation can be found here: FreeSWITCH modules for Asterisk Developers

The presentation will show you the key internal data structures and call flow when making a two-leg call in FreeSWITCH and Asterisk between SIP and PRI protocols. And of course, how to write modules for both telephony engines.

After the talk some people approached me to ask some particular questions, it seems the interest in creating FreeSWITCH modules and applications on top of it is increasing, it’s definitely going to be interesting what happens in the next years, the future of FreeSWITCH really looks promising.

http://lists.digium.com/pipermail/asterisk-dev/2009-March/037262.html (English)

http://www.saghul.net/blog/2009/03/18/sobre-el-modelo-de-desarrollo-de-asterisk/ (Spanish)

For those not involved in the Asterisk users and/or developers community, it all comes down to users complaining about Asterisk reliability and the new (1.6) development model that will allow new features to be introduced in dot releases which (they say) will make it worst. Telephony systems are damn critical. Users are used to see their computer crash (yeah, even Linux users, not that often but it happens). But telephony lines are very reliable circuits (basically because they’re very simple in nature and have been around a long time). Asterisk started a revolution and I don’t think anyone can deny that, it has brought a lot of flexibility to telephony systems, but that revolution comes of course with a cost.

I started playing with Asterisk at the beginning of 2004, I’ve seen segfaults, deadlocks and all kind of funny behaviours here and there. Sometimes a brand new release of Asterisk has basic functionality broken (like originate calls from the manager). That’s true as well, it can be said that Asterisk, out of the box, is not reliable (scalability is another beast I don’t want to talk about now), and naturally users don’t like that.

Having said that, I need to clarify what I mean by “reliable”. In this context, by reliable I mean, if you take 1.4.N release, deploy applications on top of it and then you blindly upgrade from 1.4.N to 1.4.(N+1) WITHOUT TESTING and put it in production expecting it to just work, well, good luck with that, and enjoy your new job flipping burgers at McDonalds. That is, Asterisk is not reliable for users who are not willing to put some effort when upgrading. Asterisk is a complex beast (and built on top of somewhat still shaky core), the Asterisk developers had been doing a great job improving the core, however, nasty hacks like masquerade are still there.

I don’t have problems with Asterisk being criticized, that’s what it will make it better in the end. I do have a problem though with all that people that are nothing but leeches of the community. Here is the kind of user I have a problem with:

1. They typically just ask questions in the mailing list, never spend time helping other users.
2. They download for free Asterisk and expect it to work out-of-the-box for their particular (profit) purpose without spending time not even doing their homework, testing their particular scenarios etc.
3. They bitch about Digium not fixing bugs in the bug-tracker, bugs that according to them are so damn critical to their business that they not even put a bounty on the voip-info wiki or in the asterisk-biz mailing list for someone to fix it.
4. They do not download beta releases or release candidates, they just wait for the “stable” release and again, expect things to magically work for them to profit.

In short, they don’t want to spend a single cent, nor spend some time out of their busy life, all they expect is their life to be free of problems and cash big bucks. One common argument for these users is that they are not developers and cannot help, that’s just bs, there is other ways to help and in the end you can always spend some of your free-asterisk-based business revenue to place bounties for the development of test beds or whatever you feel is needed to improve Asterisk.

At the end, I agree Asterisk development has to improve. But Digium does not has unlimited resources, and is already paying for a big team of development creating Asterisk and you can download it for free. Being free is not an excuse for lacking quality, but just think, there is limited resources and Digium is allocating those resources where it makes sense for its business. Type “core show warranty” in your Asterisk CLI and tell me what your warranty is.

In the other hand, those leech-users should know that not only users, but also some developers are unhappy with the development model (but different arguments than leech-users had). That’s why CallWeaver was born, and not only that. The biggest example I like to use of one of the foundations of open source (if you don’t like it, then fix it) is the FreeSwitch project.

The FreeSwitch project was born out of the discontent (to put it nicely) of one of the top developers of Asterisk: Anthony Minessale II, he complains a lot about Asterisk, yeah, but he also did a lot more for Asterisk than anyone else I know of beyond Mark himself and just a couple of the top developers of Asterisk. So, yes, from my perspective in some way he has earned the right to talk shit about Asterisk, because he knows it, he has proposed solutions and he has actually brought solutions: FreeSwitch. FreeSwitch has brought some serious competition to Asterisk (sorry, but let’s be serious, CallWeaver and Yate had never been close to match Asterisk functionality and usage, let alone GNU Bayonne).

FreeSwitch, from my perspective has solved many of the core problems that Asterisk has, it also solves the licensing issues (it’s MPL) and it’s developer friendly (since there is no business driving force yet behind it and you can get an svn branch right away). Probably FreeSwitch is not a short-term solution for those who are already hooked into Asterisk business, but in any case, at this point I don’t think there is short-term solutions for the problems Asterisk users complain about, it’s gonna cost ya baby, one way or another, and from my perspective that is the way is supposed to be.

In the end, Olle Johansson did quite good being the first in reply to that post and resumed the situation, which is not simple, but one thing is for sure, the community has to stop bitching and start doing. Well, I really don’t care if you keep bitching as long as you fucking do something else to make the situation better.