Project Description

The goal of the project is to implement a collection of top-level crash commands in drgn tool. The commands should provide a top-level overview for anybody who opens a kernel core dump. I plan to select a similar set of commands as seen in crash-python tool.

Goal for this Hackweek

Implement basic commands and play with the drgn internals.

Resources

  • https://crash-utility.github.io/help.html
  • https://drgn.readthedocs.io/en/latest/
  • https://crash-python.readthedocs.io/en/latest/index.html

Looking for hackers with the skills:

crash python3 kernel debug

This project is part of:

Hack Week 22

Activity

  • almost 2 years ago: marxin added keyword "crash" to this project.
  • almost 2 years ago: marxin added keyword "python3" to this project.
  • almost 2 years ago: marxin added keyword "kernel" to this project.
  • almost 2 years ago: marxin added keyword "debug" to this project.
  • almost 2 years ago: marxin started this project.
  • almost 2 years ago: marxin originated this project.

  • Comments

    • marxin
      almost 2 years ago by marxin | Reply

      I decided to implement the basic commands as part of contrib (^1) sub-folder of the project. It's the location intended for more complex listing-like (or analysis) scripts and I was able to introduce (or extend) the following commands even though my kernel knowledge is very poor. That's a good sign the drgn tool provides a friendly API and usable helper functions (^2):

      ps (extended to provide memory-related stats ^3):

      PID PPID CPU ST VMS RSS MEM% COMM 1 0 0 S 10.4M 6.5M 0.4 init 2 0 0 S 0 0 0.0 [kthreadd] ... 263 1 4 S 2.4G 163.5M 9.5 python3 264 1 5 S 2.4G 163.5M 9.5 python3 265 1 6 S 2.4G 163.5M 9.5 python3 266 1 10 S 2.4G 163.5M 9.5 python3 267 1 12 S 2.4G 163.5M 9.5 python3 268 1 13 S 2.4G 163.5M 9.5 python3 269 1 14 S 2.4G 163.5M 9.5 python3 270 1 15 S 2.4G 163.5M 9.5 python3 271 1 16 S 2.4G 163.5M 9.5 python3 ...

      sys (newly added as ^4)

      CPUS 16 DATE Fri Jan 27 20:26:24 2023 UPTIME 1 day, 7:29:37 LOAD AVERAGE 0.00, 0.00, 0.00 TASKS 317 NODENAME tw RELEASE 6.1.7-1-default VERSION #1 SMP PREEMPT_DYNAMIC Wed Jan 18 11:12:34 UTC 2023 (872045c) MACHINE x86_64 MEMORY 12.67 GiB

      vmstat (newly added ^5)

      ``` Event Count VMZONESTAT: NRFREEPAGES 512147 NRZONELRUBASE 234271 NRZONEINACTIVEANON 234271 NRZONEACTIVEANON 196 NRZONEINACTIVEFILE 97200 NRZONEACTIVEFILE 110611 NRZONEUNEVICTABLE 1000 NRZONEWRITEPENDING 84 NRMLOCK 0 NRBOUNCE 0 NRZSPAGES 0 NRFREECMAPAGES 0

      VMNODESTAT: NRLRUBASE 234322 NRINACTIVEANON 234322 NRACTIVEANON 196 NRINACTIVEFILE 97200 ... ```

      vmmap (newly added ^6)

      Start End Flgs Offset Dev Inode File path 55dee5284000-55dee53f3000 r-xp 00000000 fd:02 10515 /usr/lib/systemd/systemd 55dee53f3000-55dee5441000 r--p 0016f000 fd:02 10515 /usr/lib/systemd/systemd 55dee5441000-55dee5442000 rw-p 001bd000 fd:02 10515 /usr/lib/systemd/systemd 55dee5f4c000-55dee615d000 rw-p 00000000 00:00 0 7f5fc801c000-7f5fc8024000 r-xp 00000000 fd:02 1181379 /usr/lib64/libffi.so.7.1.0 7f5fc8024000-7f5fc8224000 ---p 00008000 fd:02 1181379 /usr/lib64/libffi.so.7.1.0 7f5fc8224000-7f5fc8225000 r--p 00008000 fd:02 1181379 /usr/lib64/libffi.so.7.1.0 ...

      mount (newly added ^7):

      Mount Type Devname Dirname ffff8fed001d8500 rootfs rootfs / ffff8fed06a197c0 proc proc /proc ffff8fed06a192c0 sysfs sysfs /sys ffff8fed06a18c80 devtmpfs devtmpfs /dev ffff8fed06a18b40 securityfs securityfs /sys/kernel/security ffff8fed06a19cc0 tmpfs tmpfs /dev/shm ffff8fed06a18500 devpts devpts /dev/pts ffff8fed06a18dc0 tmpfs tmpfs /run ...

      Existing contrib scripts

      There are other existing commands that can:

      • list TCP connections
      • list loaded kernel modules
      • list all the files on a mounted device
      • cgroup 2 listing

    • marxin
      almost 2 years ago by marxin | Reply

      When it comes to more complex verification scripts, I was able to port Vlastimil's page table walker (^1) and a verification script (^2) used for debugging of a customer bug. Please see the following git branch: ^3.

    • marxin
      almost 2 years ago by marxin | Reply

      Misc drgn observations

      • One can write scripts that work for many kernel releases. One can use symbol_name in prog technique or simple wrap a code in try ... catch block and provide a fallback for older/newer releases.
      • The project contains prebuilt vmlinux binaries for various versions (^1) and one can easily run a contrib script in QEMU for a selected Linux version:

      $ python3 -m vmtest.vm -k '5.10.*' python3 -Bm drgn contrib/ps.py Linux version 5.10.166-vmtest18.1default (drgn@drgn) (gcc (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0, GNU ld (GNU Binutils for Ubuntu) 2.34) #1 SMP Mon Feb 6 08:12:05 UTC 2023 Command line: rootfstype=9p rootflags=trans=virtio,cache=loose,msize=1048576 ro console=0,115200 panic=-1 crashkernel=256M init=/tmp/drgn-vmtest-_6sh_xhu/init x86/fpu: x87 FPU will use FXSAVE BIOS-provided physical RAM map: BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved ... PID PPID CPU ST COMM 1 0 6 S init 2 0 15 S [kthreadd] 3 2 0 I [rcu_gp] ...

    Similar Projects

    Create a DRM driver for VGA video cards by tdz

    Yes, those [VGA video cards](https://en.wikiped...


    early stage kdump support by mbrugger

    [comment]: # (Please use the project descriptio...


    Officially Become a Kernel Hacker! by m.crivellari

    Description

    My studies as well my spare tim...


    Improve various phones kernel mainline support (Qualcomm, Exynos, MediaTek) by pvorel

    Similar to previous hackweeks ( [https://hackwe...


    Contributing to Linux Kernel security by pperego

    Description

    A couple of weeks ago, I foun...