Project Description

The goal of the project is to implement a collection of top-level crash commands in drgn tool. The commands should provide a top-level overview for anybody who opens a kernel core dump. I plan to select a similar set of commands as seen in crash-python tool.

Goal for this Hackweek

Implement basic commands and play with the drgn internals.

Resources

Looking for hackers with the skills:

crash python3 kernel debug

This project is part of:

Hack Week 22

Activity

  • 10 months ago: marxin added keyword "crash" to this project.
  • 10 months ago: marxin added keyword "python3" to this project.
  • 10 months ago: marxin added keyword "kernel" to this project.
  • 10 months ago: marxin added keyword "debug" to this project.
  • 10 months ago: marxin started this project.
  • 10 months ago: marxin originated this project.

  • Comments

    • marxin
      10 months ago by marxin | Reply

      I decided to implement the basic commands as part of contrib ([^1]) sub-folder of the project. It's the location intended for more complex listing-like (or analysis) scripts and I was able to introduce (or extend) the following commands even though my kernel knowledge is very poor. That's a good sign the drgn tool provides a friendly API and usable helper functions ([^2]):

      ps (extended to provide memory-related stats [^3]):

      PID     PPID    CPU  ST    VMS     RSS  MEM%  COMM
      1       0       0    S   10.4M    6.5M   0.4 init
      2       0       0    S       0       0   0.0 [kthreadd]
      ...
      263     1       4    S    2.4G  163.5M   9.5 python3
      264     1       5    S    2.4G  163.5M   9.5 python3
      265     1       6    S    2.4G  163.5M   9.5 python3
      266     1       10   S    2.4G  163.5M   9.5 python3
      267     1       12   S    2.4G  163.5M   9.5 python3
      268     1       13   S    2.4G  163.5M   9.5 python3
      269     1       14   S    2.4G  163.5M   9.5 python3
      270     1       15   S    2.4G  163.5M   9.5 python3
      271     1       16   S    2.4G  163.5M   9.5 python3
      ...
      

      sys (newly added as [^4])

      CPUS             16
      DATE             Fri Jan 27 20:26:24 2023
      UPTIME           1 day, 7:29:37
      LOAD AVERAGE     0.00, 0.00, 0.00
      TASKS            317
      NODENAME         tw
      RELEASE          6.1.7-1-default
      VERSION          #1 SMP PREEMPT_DYNAMIC Wed Jan 18 11:12:34 UTC 2023 (872045c)
      MACHINE          x86_64
      MEMORY           12.67 GiB
      

      vmstat (newly added [^5])

      Event                                           Count
      VM_ZONE_STAT:
      NR_FREE_PAGES                                  512147
      NR_ZONE_LRU_BASE                               234271
      NR_ZONE_INACTIVE_ANON                          234271
      NR_ZONE_ACTIVE_ANON                               196
      NR_ZONE_INACTIVE_FILE                           97200
      NR_ZONE_ACTIVE_FILE                            110611
      NR_ZONE_UNEVICTABLE                              1000
      NR_ZONE_WRITE_PENDING                              84
      NR_MLOCK                                            0
      NR_BOUNCE                                           0
      NR_ZSPAGES                                          0
      NR_FREE_CMA_PAGES                                   0
      
      VM_NODE_STAT:
      NR_LRU_BASE                                    234322
      NR_INACTIVE_ANON                               234322
      NR_ACTIVE_ANON                                    196
      NR_INACTIVE_FILE                                97200
      ...
      

      vmmap (newly added [^6])

      Start        End          Flgs   Offset Dev   Inode            File path
      55dee5284000-55dee53f3000 r-xp 00000000 fd:02 10515            /usr/lib/systemd/systemd
      55dee53f3000-55dee5441000 r--p 0016f000 fd:02 10515            /usr/lib/systemd/systemd
      55dee5441000-55dee5442000 rw-p 001bd000 fd:02 10515            /usr/lib/systemd/systemd
      55dee5f4c000-55dee615d000 rw-p 00000000 00:00 0
      7f5fc801c000-7f5fc8024000 r-xp 00000000 fd:02 1181379          /usr/lib64/libffi.so.7.1.0
      7f5fc8024000-7f5fc8224000 ---p 00008000 fd:02 1181379          /usr/lib64/libffi.so.7.1.0
      7f5fc8224000-7f5fc8225000 r--p 00008000 fd:02 1181379          /usr/lib64/libffi.so.7.1.0
      ...
      

      mount (newly added [^7]):

      Mount            Type         Devname      Dirname
      ffff8fed001d8500 rootfs       rootfs       /
      ffff8fed06a197c0 proc         proc         /proc
      ffff8fed06a192c0 sysfs        sysfs        /sys
      ffff8fed06a18c80 devtmpfs     devtmpfs     /dev
      ffff8fed06a18b40 securityfs   securityfs   /sys/kernel/security
      ffff8fed06a19cc0 tmpfs        tmpfs        /dev/shm
      ffff8fed06a18500 devpts       devpts       /dev/pts
      ffff8fed06a18dc0 tmpfs        tmpfs        /run
      ...
      

      Existing contrib scripts

      There are other existing commands that can:

      • list TCP connections
      • list loaded kernel modules
      • list all the files on a mounted device
      • cgroup 2 listing

      [^1]: https://github.com/osandov/drgn/tree/main/contrib [^2]: https://drgn.readthedocs.io/en/latest/helpers.html [^3]: https://github.com/osandov/drgn/pull/257 [^4]: https://github.com/osandov/drgn/pull/256 [^5]: https://github.com/osandov/drgn/pull/252 [^6]: https://github.com/osandov/drgn/pull/263 [^7]: https://github.com/osandov/drgn/pull/251

    • marxin
      10 months ago by marxin | Reply

      When it comes to more complex verification scripts, I was able to port Vlastimil's page table walker (^1) and a verification script (^2) used for debugging of a customer bug. Please see the following git branch: ^3.

    • marxin
      10 months ago by marxin | Reply

      Misc drgn observations

      • One can write scripts that work for many kernel releases. One can use symbol_name in prog technique or simple wrap a code in try ... catch block and provide a fallback for older/newer releases.
      • The project contains prebuilt vmlinux binaries for various versions ([^1]) and one can easily run a contrib script in QEMU for a selected Linux version:
      $ python3 -m vmtest.vm -k '5.10.*' python3 -Bm drgn contrib/ps.py
      Linux version 5.10.166-vmtest18.1default (drgn@drgn) (gcc (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0, GNU ld (GNU Binutils for Ubuntu) 2.34) #1 SMP Mon Feb 6 08:12:05 UTC 2023
      Command line: rootfstype=9p rootflags=trans=virtio,cache=loose,msize=1048576 ro console=0,115200 panic=-1 crashkernel=256M init=/tmp/drgn-vmtest-_6sh_xhu/init
      x86/fpu: x87 FPU will use FXSAVE
      BIOS-provided physical RAM map:
      BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable
      BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved
      BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved
      ...
      PID     PPID    CPU  ST COMM
      1       0       6    S  init
      2       0       15   S  [kthreadd]
      3       2       0    I  [rcu_gp]
      ...
      

      [^1]: https://github.com/osandov/drgn/blob/b05e02d4ec8d8df5f96f11cc005ca821ca7e96f0/setup.py#L134-L151

    Similar Projects

    Publish two personally productive projects (preferably on github) by dmair

    Since this will be my first real "from the star...


    Cluster-Tester for SAP HANA System Replication Cluster by fmherschel

    [comment]: # (Please use the project descriptio...


    Create tool to analyze supportconfig to spot common SUSE Manager / Uyuni issues by cbosdonnat

    [comment]: # (Please use the project descriptio...


    A quantum physics experiment puzzle (designed with Google's CP-SAT solver) by moio

    [![link to video player demoing the result](htt...


    TinyTutor - an AI chat-bot powered children's tutor by dmulder

    [comment]: # (Please use the project descriptio...


    Model checking the BPF verifier by shunghsiyu

    Project Description

    BPF verifier plays a ...


    Authenticated hashes for BTRFS by dsterba

    Project Description

    Implement a checksum ...


    early stage kdump support by mbrugger

    [comment]: # (Please use the project descriptio...