Debugging MariaDB In Linux

Regardless of whether you have or don’t have a server that is compiled for debugging, there are still other ways to get more information if things go wrong. Let’s learn about debugging MariaDB in Linux in this article.

When things go wrong, it’s always good to have a version of mysqld daemon that is not stripped.

shell> file /usr/sbin/mysqld

If this doesn’t say ‘stripped’ then you are fine. If this is not the case then you should either download a binary with debugging information or compile it, without stripping the binary.

Debugging Memory Consumption Using tcmalloc

If you have a problem with a mysqld process which keeps on growing, you can use tcmalloc to investigate what is allocating memory.

Depending on the system, you have to install the tcmalloc (OpenSuse) or the google-perftools-lib (RedHat, Centos) package.

The following commands start mysqld with memory profiling and if you kill it with SIGABRT, you will get a core dump which you can check:

HEAPPROFILE=/tmp/ /usr/sbin/mysqld_safe --malloc-lib=tcmalloc --core-file-size=unlimited --core-file

Or if you prefer to invoke mysqld directly, use this:

ulimit -c unlimted
LD_PRELOAD=/usr/lib64/ HEAPPROFILE=/tmp/ /usr/sbin/mysqld --core-file

You can also add other mysqld options to the end of the above line.

Now start your client/application which uses MariaDB. You can find where memory is allocated in the /tmp/mysqld.proffile. If you find any memory issues, you can report it in the MariaDB bug tracker!

ptrace Protection and Attaching GDB to a mysqld Instance

Recent Ubuntu releases do not allow one process to check the memory of an arbitrary user’s process. As a result, when trying to attach GDB to a running MariaDB instance, you can get the following error in GDB:

ptrace: Operation not permitted

To allow GDB to attach, you needs to edit the value of the /proc/sys/kernel/yama/ptrace_scope sysctl value.

  • To modify it temporarily, open a root shell and issue:echo 0 > /proc/sys/kernel/yama/ptrace_scope
  • You can also change it permanently. For that edit as root:/etc/sysctl.d/10-ptrace.conf and set the value to 0.

Debugging a Server That Hangs

If your mysqld server hangs, you should debug it to know what happened.

Ideally, the server should be compiled for debugging, but it’s not strictly necessary:

make -j4

To know what the server is doing:

  • Find out the process number of mysqld using below command.
ps -edalf | grep mysqld
  • Attach to the process and get a backtrace using following:
gdb -p 'pid of mysqld' path-to-mysqld
set height 0
set logging file /tmp/mysqld.log
set logging on
thread apply all backtrace full

After the above, you have a full backtrace, including all local variables, in the filemysqld.log. Please note that you will only get all variables if your server is not stripped.