HOWTO extract a stack trace from teuthology (take 2)
When a Ceph teuthology integration test fails (for instance a rados jobs), it will collect core dumps which can be downloaded from the same directory where the logs and config.yaml files can be found, under the remote/mira076/coredump directory.
The binary from which the core dump comes from can be displayed with:
$ file 1425077911.7304.core ELF 64-bit LSB core file x86-64, version 1, from 'ceph-osd -f -i 3'
The teuthology logs contains command lines that can be used to install the corresponding binaries:
$ echo deb http://gitbuilder.ceph.com/ceph-deb-trusty-x86_64-basic\ /sha1/e54834bfac3c38562987730b317cb1944a96005b trusty main | \ sudo tee /etc/apt/sources.list.d/ceph.list $ sudo apt-get update $ sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes \ -o Dpkg::Options::="--force-confdef" \ -o Dpkg::Options::="--force-confold" install \ ceph=0.80.8-75-ge54834b-1trusty \ ceph-dbg=0.80.8-75-ge54834b-1trusty
The ceph-dbg package contains debug symbols that will automatically be used by gdb(1):
$ gdb /usr/bin/ceph-osd 1425077911.7304.core ... Reading symbols from /usr/bin/ceph-osd... Reading symbols from /usr/lib/debug//usr/bin/ceph-osd...done. ... [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". Core was generated by `ceph-osd -f -i 3'. Program terminated with signal SIGSEGV, Segmentation fault. #0 0x00007f59d6e9af07 in _dl_map_object_deps at dl-deps.c:528 (gdb) bt #0 0x00007f59d6e9af07 in _dl_map_object_deps at dl-deps.c:528 #1 0x00007f59d6ea1aab in dl_open_worker at dl-open.c:272 #2 0x00007f59d6e9cff4 in _dl_catch_error at dl-error.c:187 #3 0x00007f59d6ea13bb in _dl_open at dl-addr.c:61 #5 __GI__dl_addr at dl-addr.c:137 #6 0x00007f59c06dcbc0 in ?? () #7 0x00007f59d70b11c8 in _r_debug () #8 0x00007f59c06dcba0 in ?? () #9 0x00007f59c06dcbb0 in ?? () #10 0x00007f59c06dcb90 in ?? () #11 0x00007f59c06dca94 in ?? () #12 0x0000000000000000 in ?? ()