Project Description

The goal of the project is to implement a collection of top-level crash commands in drgn tool. The commands should provide a top-level overview for anybody who opens a kernel core dump. I plan to select a similar set of commands as seen in crash-python tool.

Goal for this Hackweek

Implement basic commands and play with the drgn internals.

Resources

Looking for hackers with the skills:

crash python3 kernel debug

This project is part of:

Hack Week 22

Activity

  • 10 months ago: marxin added keyword "crash" to this project.
  • 10 months ago: marxin added keyword "python3" to this project.
  • 10 months ago: marxin added keyword "kernel" to this project.
  • 10 months ago: marxin added keyword "debug" to this project.
  • 10 months ago: marxin started this project.
  • 10 months ago: marxin originated this project.

  • Comments

    • marxin
      10 months ago by marxin | Reply

      I decided to implement the basic commands as part of contrib ([^1]) sub-folder of the project. It's the location intended for more complex listing-like (or analysis) scripts and I was able to introduce (or extend) the following commands even though my kernel knowledge is very poor. That's a good sign the drgn tool provides a friendly API and usable helper functions ([^2]):

      ps (extended to provide memory-related stats [^3]):

      PID     PPID    CPU  ST    VMS     RSS  MEM%  COMM
      1       0       0    S   10.4M    6.5M   0.4 init
      2       0       0    S       0       0   0.0 [kthreadd]
      ...
      263     1       4    S    2.4G  163.5M   9.5 python3
      264     1       5    S    2.4G  163.5M   9.5 python3
      265     1       6    S    2.4G  163.5M   9.5 python3
      266     1       10   S    2.4G  163.5M   9.5 python3
      267     1       12   S    2.4G  163.5M   9.5 python3
      268     1       13   S    2.4G  163.5M   9.5 python3
      269     1       14   S    2.4G  163.5M   9.5 python3
      270     1       15   S    2.4G  163.5M   9.5 python3
      271     1       16   S    2.4G  163.5M   9.5 python3
      ...
      

      sys (newly added as [^4])

      CPUS             16
      DATE             Fri Jan 27 20:26:24 2023
      UPTIME           1 day, 7:29:37
      LOAD AVERAGE     0.00, 0.00, 0.00
      TASKS            317
      NODENAME         tw
      RELEASE          6.1.7-1-default
      VERSION          #1 SMP PREEMPT_DYNAMIC Wed Jan 18 11:12:34 UTC 2023 (872045c)
      MACHINE          x86_64
      MEMORY           12.67 GiB
      

      vmstat (newly added [^5])

      Event                                           Count
      VM_ZONE_STAT:
      NR_FREE_PAGES                                  512147
      NR_ZONE_LRU_BASE                               234271
      NR_ZONE_INACTIVE_ANON                          234271
      NR_ZONE_ACTIVE_ANON                               196
      NR_ZONE_INACTIVE_FILE                           97200
      NR_ZONE_ACTIVE_FILE                            110611
      NR_ZONE_UNEVICTABLE                              1000
      NR_ZONE_WRITE_PENDING                              84
      NR_MLOCK                                            0
      NR_BOUNCE                                           0
      NR_ZSPAGES                                          0
      NR_FREE_CMA_PAGES                                   0
      
      VM_NODE_STAT:
      NR_LRU_BASE                                    234322
      NR_INACTIVE_ANON                               234322
      NR_ACTIVE_ANON                                    196
      NR_INACTIVE_FILE                                97200
      ...
      

      vmmap (newly added [^6])

      Start        End          Flgs   Offset Dev   Inode            File path
      55dee5284000-55dee53f3000 r-xp 00000000 fd:02 10515            /usr/lib/systemd/systemd
      55dee53f3000-55dee5441000 r--p 0016f000 fd:02 10515            /usr/lib/systemd/systemd
      55dee5441000-55dee5442000 rw-p 001bd000 fd:02 10515            /usr/lib/systemd/systemd
      55dee5f4c000-55dee615d000 rw-p 00000000 00:00 0
      7f5fc801c000-7f5fc8024000 r-xp 00000000 fd:02 1181379          /usr/lib64/libffi.so.7.1.0
      7f5fc8024000-7f5fc8224000 ---p 00008000 fd:02 1181379          /usr/lib64/libffi.so.7.1.0
      7f5fc8224000-7f5fc8225000 r--p 00008000 fd:02 1181379          /usr/lib64/libffi.so.7.1.0
      ...
      

      mount (newly added [^7]):

      Mount            Type         Devname      Dirname
      ffff8fed001d8500 rootfs       rootfs       /
      ffff8fed06a197c0 proc         proc         /proc
      ffff8fed06a192c0 sysfs        sysfs        /sys
      ffff8fed06a18c80 devtmpfs     devtmpfs     /dev
      ffff8fed06a18b40 securityfs   securityfs   /sys/kernel/security
      ffff8fed06a19cc0 tmpfs        tmpfs        /dev/shm
      ffff8fed06a18500 devpts       devpts       /dev/pts
      ffff8fed06a18dc0 tmpfs        tmpfs        /run
      ...
      

      Existing contrib scripts

      There are other existing commands that can:

      • list TCP connections
      • list loaded kernel modules
      • list all the files on a mounted device
      • cgroup 2 listing

      [^1]: https://github.com/osandov/drgn/tree/main/contrib [^2]: https://drgn.readthedocs.io/en/latest/helpers.html [^3]: https://github.com/osandov/drgn/pull/257 [^4]: https://github.com/osandov/drgn/pull/256 [^5]: https://github.com/osandov/drgn/pull/252 [^6]: https://github.com/osandov/drgn/pull/263 [^7]: https://github.com/osandov/drgn/pull/251

    • marxin
      10 months ago by marxin | Reply

      When it comes to more complex verification scripts, I was able to port Vlastimil's page table walker (^1) and a verification script (^2) used for debugging of a customer bug. Please see the following git branch: ^3.

    • marxin
      10 months ago by marxin | Reply

      Misc drgn observations

      • One can write scripts that work for many kernel releases. One can use symbol_name in prog technique or simple wrap a code in try ... catch block and provide a fallback for older/newer releases.
      • The project contains prebuilt vmlinux binaries for various versions ([^1]) and one can easily run a contrib script in QEMU for a selected Linux version:
      $ python3 -m vmtest.vm -k '5.10.*' python3 -Bm drgn contrib/ps.py
      Linux version 5.10.166-vmtest18.1default (drgn@drgn) (gcc (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0, GNU ld (GNU Binutils for Ubuntu) 2.34) #1 SMP Mon Feb 6 08:12:05 UTC 2023
      Command line: rootfstype=9p rootflags=trans=virtio,cache=loose,msize=1048576 ro console=0,115200 panic=-1 crashkernel=256M init=/tmp/drgn-vmtest-_6sh_xhu/init
      x86/fpu: x87 FPU will use FXSAVE
      BIOS-provided physical RAM map:
      BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable
      BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved
      BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved
      ...
      PID     PPID    CPU  ST COMM
      1       0       6    S  init
      2       0       15   S  [kthreadd]
      3       2       0    I  [rcu_gp]
      ...
      

      [^1]: https://github.com/osandov/drgn/blob/b05e02d4ec8d8df5f96f11cc005ca821ca7e96f0/setup.py#L134-L151

    Similar Projects

    Publish two personally productive projects (preferably on github) by dmair

    Since this will be my first real "from the star...


    Create tool to analyze supportconfig to spot common SUSE Manager / Uyuni issues by cbosdonnat

    [comment]: # (Please use the project descriptio...


    A quantum physics experiment puzzle (designed with Google's CP-SAT solver) by moio

    [![link to video player demoing the result](htt...


    Cluster-Tester for SAP HANA System Replication Cluster by fmherschel

    [comment]: # (Please use the project descriptio...


    TinyTutor - an AI chat-bot powered children's tutor by dmulder

    [comment]: # (Please use the project descriptio...


    Model checking the BPF verifier by shunghsiyu

    Project Description

    BPF verifier plays a ...


    early stage kdump support by mbrugger

    [comment]: # (Please use the project descriptio...


    Authenticated hashes for BTRFS by dsterba

    Project Description

    Implement a checksum ...