Tuesday, July 24, 2007

Debugging on Solaris

About Crash Dump File:
unix.<x> - Symbol Tables
vmcore.<x> - Memory Dump
bounds - Contains the sequence number to use for the next execution of savecore

Debugging with MDB


Debugging with SCAT (SUNWscat)
By default Solaris CAT is started in Read-only mode but it can be started with write privileges by using '--write' flag.

Command: dumpinfo
Description: Provides brief overview of crash dump.
How to use: /opt/SUNWscat/bin/dumpinfo ./vmcore.0
Sample Output:
Dumphdr_version=9
Dump_magic=defec8ed
Dump_pagesize=8192
Dump_pageshift=13
Dump_platform="SUNW,Ultra-60"
Dump_panicstring="sync initiated"
Dump_wordsize=64
Dump_version="Generic_108528-29"
Dump_release="5.8"
Dump_machine="sun4u"


Command: dumphdr
Description: Provides information about crash header.
How to use: /opt/SUNWscat/bin/dumphdr ./vmcore.0
Sample Output:
magic: defec8ed
version: 9
flags: 0x2 (INVALID|COMPLETE)
wordsize: 64
start: 215220224
ksyms: 8192
pfn: 925696
map: 1007616
data: 1662976
utsname.sysname: SunOS
utsname.nodename: sunultrafour.delhi.solidcore.local
utsname.release: 5.8
utsname.version: Generic_108528-29
utsname.machine: sun4u
platform: SUNW,Ultra-60
panicstr: sync initiated
crashtime: Tue Jul 24 17:21:02 EDT 2007
pageshift: 13
pagesize: 8192 (8K)
hashmask: 0x3fff
nvtop: 9821
npages: 9812 (76.0M)
ksyms_size: 912693 (891K)
ksyms_csize: 912693 (891K)
calc/actual size: 82042880 (78.0M)


How to invoke scat?
  • Debugging crash dump: /opt/SUNWscat/bin/scat <number of crash> [Example for vmcore.0 it will be <0>]. This will provide SolarisCAT prompt. From here rest of the commands are used.
  • Debugging running kernel: /opt/SUNWscat/bin/scat. Will provide SolarisCAT(live)> similar kind of prompt

How to view stack of panic thread?
'thread' or 'panic' provides stack trace of thread which was on CPU at the time of crash or the thread that initiated crash dump.

How to view stack of all the threads?
'stack summary' provides a concise stack of all the threads in the system at the time of crash.
'tlist' provides detailed stack with pid, registers, function parameters, cpu, etc etc.

How to see 'msgbuf'?
'msgbuf'

How to dump value of some global variable or of some memory address?
'rd' reads one word from memory. On 32-bit Machine 4 bytes and on 64-bit machine 8 bytes.
'rd32' reads 4 bytes of data.
'rd64' reads 8 bytes of data.
'rds' shows string data.
'rdh' dumps half word of memory (rdh <addr> <count>)

Examples:
rd32 lwp_default_stksize [output: 0x10470394 genunix(bss):lwp_default_stksize+0x0 = 0x4000]
rd rootvp [output: 0x1046e238 genunix(bss):rootvp+0x0 = 0x30000ed9e10]

How to dump memory at some address and show corresponding structure?
'sdump'

Example:
sdump 0x1046e238 vnode - Prints vnode structure for rootvp

{
v_lock = {
_opaque = [ 0x30000ed9e10 ]
}
v_flag = 0x0
v_count = 0x166000
v_vfsmountedhere = 0x30000067508
v_op = NULL
v_vfsp = 0x30000085ce8
v_stream = NULL
v_pages = NULL
v_type = 0 (VNON)
v_rdev = 0x1500000018 (21(log),24)
v_data = 0xe00000003
v_filocks = 0x1800000007
v_shrlocks = 0x6b00000000
v_cv = {
_opaque = 0x0
}
v_locality = 0x4a4
}

How to see fields of a structure?
'stype'

Example:
stype vnode

struct vnode { (size: 0x70 bytes)
typedef kmutex_t = struct mutex { (size: 0x8 bytes)
void *[0x1] _opaque; (offset 0x0 bytes, size 0x8 bytes)
} v_lock; (offset 0x0 bytes, size 0x8 bytes)
typedef ushort_t = unsigned short v_flag; (offset 0x8 bytes, size 0x2 bytes)
typedef uint_t = unsigned int v_count; (offset 0xc bytes, size 0x4 bytes)
struct vfs *v_vfsmountedhere; (offset 0x10 bytes, size 0x8 bytes)
struct vnodeops *v_op; (offset 0x18 bytes, size 0x8 bytes)
struct vfs *v_vfsp; (offset 0x20 bytes, size 0x8 bytes)
struct stdata *v_stream; (offset 0x28 bytes, size 0x8 bytes)
struct page *v_pages; (offset 0x30 bytes, size 0x8 bytes)
enum vtype {VNON=0, VREG=1, VDIR=2, VBLK=3, VCHR=4, VLNK=5, VFIFO=6, VDOOR=7, VPROC=8, VSOCK=9, VBAD=10} v_type; (offset 0x38 bytes, size 0x4 bytes)
typedef dev_t = unsigned long/long long v_rdev; (offset 0x40 bytes, size 0x8 bytes)
typedef caddr_t = char *v_data; (offset 0x48 bytes, size 0x8 bytes)
struct filock *v_filocks; (offset 0x50 bytes, size 0x8 bytes)
struct shrlocklist *v_shrlocks; (offset 0x58 bytes, size 0x8 bytes)
typedef kcondvar_t = struct _kcondvar { (size: 0x2 bytes)
typedef ushort_t = unsigned short _opaque; (offset 0x60 bytes, size 0x2 bytes)
} v_cv; (offset 0x60 bytes, size 0x2 bytes)
void *v_locality; (offset 0x68 bytes, size 0x8 bytes)
} ;

How to get assembly instruction for some address?
"rdi" dumps memory in instruction format starting for <addr> for the specified <count>. If option
  • '-f' - the entire function that the <addr> is in displayed.
  • '-b' - the entire function up to <addr> is displayed.
  • '-m' - specified modules entire text segment is displayed.
  • '-p' - <count> instructions before <addr> are displayed.

Writing Kernel Memory Variables on the fly?
'scat --write' (opens kernel with write privileges)

'wr' - write word (example: wr memscrub_verbose 1)
'wr hword' - Half word (wr hword memscrub_verbose 1)

Querying size of some type?
calc "sizeof(<data type>)" - example calc "sizeof(uint_t)"

How to see frame's data?
'frame <pc>'
Lets consider following stack trace:
<leaf trap>genunix:strcmp+0x8 (%o:0x7813b888, 0x0, 0x3c, 0x2, 0x2, 0x10436800)
calkm:sc_getvfsswbyname+0x88 (0x10436b08, 0x27, 0x7813b9e0, 0x7813b8c8, 0x780e4000, 0x0)
calkm:hook_vfsops+0xf8 (0x780e4090, 0x0, 0x780e4088, 0x7813b800, 0x7813b800, 0x7813b800)
calkm:sc_dwrite_init+0xf0 (0x5, 0x10432298, 0x10434298, 0x78129900, 0x780e4000, 0x0)
calkm:_init+0xa0 (0x0, 0x7811a000, 0x0, 0x0, 0x30001efaf98, 0x300000328b0)
genunix:modinstall+0x144 (0x10437c00, 0x40000000, 0x300021c2820, 0x7811a000, 0x26, 0x0)
genunix:mod_hold_installed_mod+0x4c (0x300021c2820, 0x2a1002f1a3c, 0x0, 0x0, 0x300021c2820, 0x0)

'frame sc_getvfsswbyname' will provide data for this frame
'frame hook_vfsops' will provide data for next frame.

How to dump data starting at some memory location?
'mdump' - example 'mdump <addr> <count>

How to search for all possible stacks?
'findstk' - finds all possible stacks
  • '-m' - reports only stack containing module
  • '-c' - reports only stack containing function

Web Links:
http://cmynhier.blogspot.com/2006/11/unlinked-files-and-mdb.html
http://prefetch.net/blog/index.php/2006/03/13/finding-out-the-file-name-associated-with-an-inode/
http://blogs.sun.com/jwadams/entry/debugging_with_libumem_and_mdb
http://blogs.sun.com/bmc/date/20050125
http://blogs.sun.com/eschrock/category/Solaris
http://docs.sun.com/source/817-3886-10/chap06.html





1 comment:

Manoj Awasthi said...

no posts for long. hibernation?