How to get the reason for a crash of your Kamailio (OpenSER) server?
After a crash of the server it will generate a core file in its working directory, if the functionality is not disabled. This core file can be used to get the cause (or at least a hint) of the crash. The core file contains the memory of the process at the moment it crashes, so it could contain sensitive data.
Core files can be examined with gdb, the gnu debugger. It can read the crash informations, and display (among other things) the backtrace that leads to the crash. Just run:
gdb $kamailio_binary $core_file
(Substitute the variables with the information in your installation.) After gdb finished to read its input type backtrace, gdb will then output the backtrace. This information should added to your bug report of the crash. Please don't delete the core files after the backtrace generation, as its really helpful for eventual further investigations.
In order to get a meaningful backtrace gdb needs some debugging informations in its binary. If you compiled from the source, no additional steps should be necessary from you. If you install a binary package, e.g. from debian, you need to install the -dbg package too.
If you don't understand the information that gdb prints, some developer on the mailling list should be able to help you.
Sometimes the server don't write the core file after a crash. INFO:core:handle_sigs: core was not generated
First check if the “disable_core_dump” variable is set in your configuration. If you run the server from a debian package, check the DUMP_CORE setting in the “/etc/default/kamailio” file. If this are all not set, then there is probably some ulimit configured that prohibits the core file generation. Just run:
ulimit -c unlimited
in the shell where you started the server. If you run from a init script it should normally already take care of this. If its still not create a core file then perhaps the server has no permissions to write into its working directory, check its permissions and the startup option “-w”..
To get per-process core file then you can use on Linux:
echo "1" > /proc/sys/kernel/core_uses_pid