drgn: implement crash top-level commands
a project by marxin
Updated 10 months ago. No love. 2 followers.

Project Description

The goal of the project is to implement a collection of top-level crash commands in drgn tool. The commands should provide a top-level overview for anybody who opens a kernel core dump. I plan to select a similar set of commands as seen in crash-python tool.

Goal for this Hackweek

Implement basic commands and play with the drgn internals.

Resources

Join this project

Looking for hackers with the skills:

crash python3 kernel debug

This project is part of:

Hack Week 22

Activity

  • 10 months ago: marxin added keyword "crash" to this project.
  • 10 months ago: marxin added keyword "python3" to this project.
  • 10 months ago: marxin added keyword "kernel" to this project.
  • 10 months ago: marxin added keyword "debug" to this project.
  • 10 months ago: marxin started this project.
  • 10 months ago: marxin originated this project.


    • Comments

    • marxin
      10 months ago by marxin | Reply

      I decided to implement the basic commands as part of contrib ([^1]) sub-folder of the project. It's the location intended for more complex listing-like (or analysis) scripts and I was able to introduce (or extend) the following commands even though my kernel knowledge is very poor. That's a good sign the drgn tool provides a friendly API and usable helper functions ([^2]):

      ps (extended to provide memory-related stats [^3]):

      PID     PPID    CPU  ST    VMS     RSS  MEM%  COMM
1       0       0    S   10.4M    6.5M   0.4 init
2       0       0    S       0       0   0.0 [kthreadd]
...
263     1       4    S    2.4G  163.5M   9.5 python3
264     1       5    S    2.4G  163.5M   9.5 python3
265     1       6    S    2.4G  163.5M   9.5 python3
266     1       10   S    2.4G  163.5M   9.5 python3
267     1       12   S    2.4G  163.5M   9.5 python3
268     1       13   S    2.4G  163.5M   9.5 python3
269     1       14   S    2.4G  163.5M   9.5 python3
270     1       15   S    2.4G  163.5M   9.5 python3
271     1       16   S    2.4G  163.5M   9.5 python3
...

      sys (newly added as [^4])

      CPUS             16
DATE             Fri Jan 27 20:26:24 2023
UPTIME           1 day, 7:29:37
LOAD AVERAGE     0.00, 0.00, 0.00
TASKS            317
NODENAME         tw
RELEASE          6.1.7-1-default
VERSION          #1 SMP PREEMPT_DYNAMIC Wed Jan 18 11:12:34 UTC 2023 (872045c)
MACHINE          x86_64
MEMORY           12.67 GiB

      vmstat (newly added [^5])

      Event                                           Count
VM_ZONE_STAT:
NR_FREE_PAGES                                  512147
NR_ZONE_LRU_BASE                               234271
NR_ZONE_INACTIVE_ANON                          234271
NR_ZONE_ACTIVE_ANON                               196
NR_ZONE_INACTIVE_FILE                           97200
NR_ZONE_ACTIVE_FILE                            110611
NR_ZONE_UNEVICTABLE                              1000
NR_ZONE_WRITE_PENDING                              84
NR_MLOCK                                            0
NR_BOUNCE                                           0
NR_ZSPAGES                                          0
NR_FREE_CMA_PAGES                                   0

VM_NODE_STAT:
NR_LRU_BASE                                    234322
NR_INACTIVE_ANON                               234322
NR_ACTIVE_ANON                                    196
NR_INACTIVE_FILE                                97200
...

      vmmap (newly added [^6])

      Start        End          Flgs   Offset Dev   Inode            File path
55dee5284000-55dee53f3000 r-xp 00000000 fd:02 10515            /usr/lib/systemd/systemd
55dee53f3000-55dee5441000 r--p 0016f000 fd:02 10515            /usr/lib/systemd/systemd
55dee5441000-55dee5442000 rw-p 001bd000 fd:02 10515            /usr/lib/systemd/systemd
55dee5f4c000-55dee615d000 rw-p 00000000 00:00 0
7f5fc801c000-7f5fc8024000 r-xp 00000000 fd:02 1181379          /usr/lib64/libffi.so.7.1.0
7f5fc8024000-7f5fc8224000 ---p 00008000 fd:02 1181379          /usr/lib64/libffi.so.7.1.0
7f5fc8224000-7f5fc8225000 r--p 00008000 fd:02 1181379          /usr/lib64/libffi.so.7.1.0
...

      mount (newly added [^7]):

      Mount            Type         Devname      Dirname
ffff8fed001d8500 rootfs       rootfs       /
ffff8fed06a197c0 proc         proc         /proc
ffff8fed06a192c0 sysfs        sysfs        /sys
ffff8fed06a18c80 devtmpfs     devtmpfs     /dev
ffff8fed06a18b40 securityfs   securityfs   /sys/kernel/security
ffff8fed06a19cc0 tmpfs        tmpfs        /dev/shm
ffff8fed06a18500 devpts       devpts       /dev/pts
ffff8fed06a18dc0 tmpfs        tmpfs        /run
...

      Existing contrib scripts

      There are other existing commands that can:

      • list TCP connections
      • list loaded kernel modules
      • list all the files on a mounted device
      • cgroup 2 listing

      [^1]: https://github.com/osandov/drgn/tree/main/contrib [^2]: https://drgn.readthedocs.io/en/latest/helpers.html [^3]: https://github.com/osandov/drgn/pull/257 [^4]: https://github.com/osandov/drgn/pull/256 [^5]: https://github.com/osandov/drgn/pull/252 [^6]: https://github.com/osandov/drgn/pull/263 [^7]: https://github.com/osandov/drgn/pull/251

    • marxin
      10 months ago by marxin | Reply

      When it comes to more complex verification scripts, I was able to port Vlastimil's page table walker (^1) and a verification script (^2) used for debugging of a customer bug. Please see the following git branch: ^3.

    • marxin
      10 months ago by marxin | Reply

      Misc drgn observations

      • One can write scripts that work for many kernel releases. One can use symbol_name in prog technique or simple wrap a code in try ... catch block and provide a fallback for older/newer releases.
      • The project contains prebuilt vmlinux binaries for various versions ([^1]) and one can easily run a contrib script in QEMU for a selected Linux version:
      $ python3 -m vmtest.vm -k '5.10.*' python3 -Bm drgn contrib/ps.py
Linux version 5.10.166-vmtest18.1default (drgn@drgn) (gcc (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0, GNU ld (GNU Binutils for Ubuntu) 2.34) #1 SMP Mon Feb 6 08:12:05 UTC 2023
Command line: rootfstype=9p rootflags=trans=virtio,cache=loose,msize=1048576 ro console=0,115200 panic=-1 crashkernel=256M init=/tmp/drgn-vmtest-_6sh_xhu/init
x86/fpu: x87 FPU will use FXSAVE
BIOS-provided physical RAM map:
BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable
BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved
BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved
...
PID     PPID    CPU  ST COMM
1       0       6    S  init
2       0       15   S  [kthreadd]
3       2       0    I  [rcu_gp]
...

      [^1]: https://github.com/osandov/drgn/blob/b05e02d4ec8d8df5f96f11cc005ca821ca7e96f0/setup.py#L134-L151

    Similar Projects

    Publish two personally productive projects (preferably on github) by dmair

    Since this will be my first real "from the star...

    Create tool to analyze supportconfig to spot common SUSE Manager / Uyuni issues by cbosdonnat

    [comment]: # (Please use the project descriptio...

    A quantum physics experiment puzzle (designed with Google's CP-SAT solver) by moio

    [![link to video player demoing the result](htt...

    Cluster-Tester for SAP HANA System Replication Cluster by fmherschel

    [comment]: # (Please use the project descriptio...

    TinyTutor - an AI chat-bot powered children's tutor by dmulder

    [comment]: # (Please use the project descriptio...

    Model checking the BPF verifier by shunghsiyu

    Project Description

    BPF verifier plays a ...

    early stage kdump support by mbrugger

    [comment]: # (Please use the project descriptio...

    Authenticated hashes for BTRFS by dsterba

    Project Description

    Implement a checksum ...