drgn: implement crash top-level commands
a project by marxin
Updated 8 months ago. No love. 2 followers.

Project Description

The goal of the project is to implement a collection of top-level crash commands in drgn tool. The commands should provide a top-level overview for anybody who opens a kernel core dump. I plan to select a similar set of commands as seen in crash-python tool.

Goal for this Hackweek

Implement basic commands and play with the drgn internals.

Resources

Join this project

Looking for hackers with the skills:

crash python3 kernel debug

This project is part of:

Hack Week 22

Activity

  • 8 months ago: marxin added keyword "crash" to this project.
  • 8 months ago: marxin added keyword "python3" to this project.
  • 8 months ago: marxin added keyword "kernel" to this project.
  • 8 months ago: marxin added keyword "debug" to this project.
  • 8 months ago: marxin started this project.
  • 8 months ago: marxin originated this project.


    • Comments

    • marxin
      8 months ago by marxin | Reply

      I decided to implement the basic commands as part of contrib ([^1]) sub-folder of the project. It's the location intended for more complex listing-like (or analysis) scripts and I was able to introduce (or extend) the following commands even though my kernel knowledge is very poor. That's a good sign the drgn tool provides a friendly API and usable helper functions ([^2]):

      ps (extended to provide memory-related stats [^3]):

      PID     PPID    CPU  ST    VMS     RSS  MEM%  COMM
1       0       0    S   10.4M    6.5M   0.4 init
2       0       0    S       0       0   0.0 [kthreadd]
...
263     1       4    S    2.4G  163.5M   9.5 python3
264     1       5    S    2.4G  163.5M   9.5 python3
265     1       6    S    2.4G  163.5M   9.5 python3
266     1       10   S    2.4G  163.5M   9.5 python3
267     1       12   S    2.4G  163.5M   9.5 python3
268     1       13   S    2.4G  163.5M   9.5 python3
269     1       14   S    2.4G  163.5M   9.5 python3
270     1       15   S    2.4G  163.5M   9.5 python3
271     1       16   S    2.4G  163.5M   9.5 python3
...

      sys (newly added as [^4])

      CPUS             16
DATE             Fri Jan 27 20:26:24 2023
UPTIME           1 day, 7:29:37
LOAD AVERAGE     0.00, 0.00, 0.00
TASKS            317
NODENAME         tw
RELEASE          6.1.7-1-default
VERSION          #1 SMP PREEMPT_DYNAMIC Wed Jan 18 11:12:34 UTC 2023 (872045c)
MACHINE          x86_64
MEMORY           12.67 GiB

      vmstat (newly added [^5])

      Event                                           Count
VM_ZONE_STAT:
NR_FREE_PAGES                                  512147
NR_ZONE_LRU_BASE                               234271
NR_ZONE_INACTIVE_ANON                          234271
NR_ZONE_ACTIVE_ANON                               196
NR_ZONE_INACTIVE_FILE                           97200
NR_ZONE_ACTIVE_FILE                            110611
NR_ZONE_UNEVICTABLE                              1000
NR_ZONE_WRITE_PENDING                              84
NR_MLOCK                                            0
NR_BOUNCE                                           0
NR_ZSPAGES                                          0
NR_FREE_CMA_PAGES                                   0

VM_NODE_STAT:
NR_LRU_BASE                                    234322
NR_INACTIVE_ANON                               234322
NR_ACTIVE_ANON                                    196
NR_INACTIVE_FILE                                97200
...

      vmmap (newly added [^6])

      Start        End          Flgs   Offset Dev   Inode            File path
55dee5284000-55dee53f3000 r-xp 00000000 fd:02 10515            /usr/lib/systemd/systemd
55dee53f3000-55dee5441000 r--p 0016f000 fd:02 10515            /usr/lib/systemd/systemd
55dee5441000-55dee5442000 rw-p 001bd000 fd:02 10515            /usr/lib/systemd/systemd
55dee5f4c000-55dee615d000 rw-p 00000000 00:00 0
7f5fc801c000-7f5fc8024000 r-xp 00000000 fd:02 1181379          /usr/lib64/libffi.so.7.1.0
7f5fc8024000-7f5fc8224000 ---p 00008000 fd:02 1181379          /usr/lib64/libffi.so.7.1.0
7f5fc8224000-7f5fc8225000 r--p 00008000 fd:02 1181379          /usr/lib64/libffi.so.7.1.0
...

      mount (newly added [^7]):

      Mount            Type         Devname      Dirname
ffff8fed001d8500 rootfs       rootfs       /
ffff8fed06a197c0 proc         proc         /proc
ffff8fed06a192c0 sysfs        sysfs        /sys
ffff8fed06a18c80 devtmpfs     devtmpfs     /dev
ffff8fed06a18b40 securityfs   securityfs   /sys/kernel/security
ffff8fed06a19cc0 tmpfs        tmpfs        /dev/shm
ffff8fed06a18500 devpts       devpts       /dev/pts
ffff8fed06a18dc0 tmpfs        tmpfs        /run
...

      Existing contrib scripts

      There are other existing commands that can:

      • list TCP connections
      • list loaded kernel modules
      • list all the files on a mounted device
      • cgroup 2 listing

      [^1]: https://github.com/osandov/drgn/tree/main/contrib [^2]: https://drgn.readthedocs.io/en/latest/helpers.html [^3]: https://github.com/osandov/drgn/pull/257 [^4]: https://github.com/osandov/drgn/pull/256 [^5]: https://github.com/osandov/drgn/pull/252 [^6]: https://github.com/osandov/drgn/pull/263 [^7]: https://github.com/osandov/drgn/pull/251

    • marxin
      8 months ago by marxin | Reply

      When it comes to more complex verification scripts, I was able to port Vlastimil's page table walker (^1) and a verification script (^2) used for debugging of a customer bug. Please see the following git branch: ^3.

    • marxin
      8 months ago by marxin | Reply

      Misc drgn observations

      • One can write scripts that work for many kernel releases. One can use symbol_name in prog technique or simple wrap a code in try ... catch block and provide a fallback for older/newer releases.
      • The project contains prebuilt vmlinux binaries for various versions ([^1]) and one can easily run a contrib script in QEMU for a selected Linux version:
      $ python3 -m vmtest.vm -k '5.10.*' python3 -Bm drgn contrib/ps.py
Linux version 5.10.166-vmtest18.1default (drgn@drgn) (gcc (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0, GNU ld (GNU Binutils for Ubuntu) 2.34) #1 SMP Mon Feb 6 08:12:05 UTC 2023
Command line: rootfstype=9p rootflags=trans=virtio,cache=loose,msize=1048576 ro console=0,115200 panic=-1 crashkernel=256M init=/tmp/drgn-vmtest-_6sh_xhu/init
x86/fpu: x87 FPU will use FXSAVE
BIOS-provided physical RAM map:
BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable
BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved
BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved
...
PID     PPID    CPU  ST COMM
1       0       6    S  init
2       0       15   S  [kthreadd]
3       2       0    I  [rcu_gp]
...

      [^1]: https://github.com/osandov/drgn/blob/b05e02d4ec8d8df5f96f11cc005ca821ca7e96f0/setup.py#L134-L151

    Similar Projects

    early stage kdump support by mbrugger

    [comment]: # (Please use the project descriptio...