L3: Improve crash-setup, develop a core-setup
a project by sparschauer
Updated almost 3 years ago. 4 hacker ♥️.

The current crash-setup source is located here. Pretty much is working nicely but it doesn't care for the debug source making it impossible to use the crash> gdb list * command right away. This is bug 997558 which should be worked on.

A further script would be really useful to build up the analysis environment on l3slave for core dumps of crashed user-space applications. The MF Open Enterprise Server (OES) has a tool called NetIQ GetCore Utility which prepares such an environment with files opencore.sh, opencore.ini, the binaries of the executable and libs, and the core dump itself. The files it generates can be used as a base and prove that it is possible to build up a core dump analysis environment decoupled from the host system. What it is missing is setting up the debug symbols and the debug source as well. I've done that manually already. So this is possible and should be automated.

Both these topics are efficiency improvement tasks for the L3 work.

Join this project

Looking for hackers with the skills:

shell c

This project is part of:

Hack Week 17

Activity

  • over 6 years ago: mkubecek liked this project.
  • over 6 years ago: sparschauer liked this project.
  • over 6 years ago: sparschauer added keyword "shell" to this project.
  • over 6 years ago: sparschauer added keyword "c" to this project.
  • over 6 years ago: mkoutny liked this project.
  • over 6 years ago: mbrugger liked this project.
  • over 6 years ago: sparschauer started this project.
  • over 6 years ago: sparschauer originated this project.


    • Comments

    • sparschauer
      over 6 years ago by sparschauer | Reply

      The core-setup project should not depend on l3slave and should work with openSUSE for non-employees as well. It should have different modes for:

      • analysis on the system which collected the dump:
        • auto-get debuginfo and debugsource packages with zypper - packages are latest? (recommended for reproducible segfaults)
        • include paths to custom code like e.g. scanmem which I compile from source as an upstream maintainer
      • analysis on another system
        • auto-get and extract RPMs from SUSE servers like e.g. http://download.opensuse.org/update/leap/42.3/oss/x86_64/, http://download.opensuse.org/debug/update/leap/42.3/oss/x86_64/
      • SLES
        • get packages from NFS mounts
        • get packages from *.suse.de servers
      • openSUSE
        • get packages from download.opensuse.org or mirrors

      Key features:

      • read coredump, detect crashed process and libs
      • convert build-id to package name + version with the help of SUSE repos
      • maybe use rpm.txt from supportconfig if exact package info cannot be gathered from the dump
      • get and extract the required binary, debuginfo, and debugsource packages
      • build up opencore.sh and opencore.ini - maybe with support for custom settings

      Initial tasks:

      • check for, evaluate, and analyze similar tools
      • analyze coredump structure - What is included and can be used?

      Main goals:

      • prevent gdb from picking up wrong code/source/debuginfo or wrong packages
      • take needed code parts from other FOSS projects to avoid duplicate efforts

    • sparschauer
      over 6 years ago by sparschauer | Reply

      For crash-setup -d vmcore, crash-setup uses kdumpid which uses libkdumpfile.

      Example output of kdumpid:

      Format: compressed kdump
      Arch: x86_64
      Version: 4.4.138-94.39-default

      • sparschauer
        over 6 years ago by sparschauer | Reply

        crash-setup uses kernel-source.git to convert the version into a git tag, then to a commit, and then to the oldest git branch containing it. This step is extremely slow as it is done by comparing the commit id in the git log of all the configured git branches from old to new. The SLE release is gathered from this. I'm sure this can be sped up.

        Example: 4.4.138-94.39 -> rpm-4.4.138-94.39 -> baa07f9df91b -> SLE12-SP3

        • sparschauer
          over 6 years ago by sparschauer | Reply

          git describe is even slower. Only configuring major.minor per branch would help.

    • sparschauer
      over 6 years ago by sparschauer | Reply

      I've introduced a new crash-setup option '-s' or '--source' to get the debugsource package automatically as well. With the crash cd command I can get to the right directory so that the gdb list command will work. The idea is to create a file opencrash.sh containing crash -i ./opencrash.ini vmlinux.gz vmlinux.debug vmcore and a file opencrash.ini containing e.g. cd ./root/usr/src/debug/kernel-default-4.4.138/linux-4.4/kernel automatically.

      • sparschauer
        over 6 years ago by sparschauer | Reply

        Autocreation of opencrash.sh/.ini implemented. Since SLE12 there are different paths to source files. So cd root is used for SLE11 and before.

        Commits can be found here.

    • sparschauer
      over 6 years ago by sparschauer | Reply

      Merge request for crash-setup submitted.

    • sparschauer
      over 6 years ago by sparschauer | Reply

      I worked on the core-setup already. gdb -ex "quit" -c core can get the build id of the main program running. Then we can look into repodata/*primary.xml.gz and search for it. The next "location href" line above shows the name of the debuginfo package.

      Example:
      $ ./core-setup ./core.hald.2341
      main build id from core.hald.2341: da45f02baff8ef519840d5a17fd15926f2c802e2
      looking up main build id...
      x86_64/hal-debuginfo-0.5.12-23.76.1.x86_64.rpm

      Now we can remove "-debuginfo" from the RPM name and get the standard package name "hal-0.5.12-23.76.1.x86_64.rpm". With these two packages it is possible to run gdb regularly and to gather the build ids of the libraries.

    • sparschauer
      over 6 years ago by sparschauer | Reply

      I made a lot of progress with core-setup. add-emoji It handles the hald example core dump pretty well already. The CPU arch, the crashed executable, and the main build id are gathered from gdb -batch -ex "info auxv" -c core. Besides the packages for the crashed executable, the debug and standard packages for 20 of 22 libraries are fetched and extracted.

      Unfortunately, just removing "-debuginfo" from the package name is not enough. There are more standard packages required for the same debuginfo package. The biggest issue is that the build id is not stored in the coredump and only reflects the standard packages of the host system. So the wrong package versions are gathered.

      Looks like I have to parse rpm.txt from the supportconfig for the right versions and check which package provides the library file which has to be extracted first and then make sure that really the correct files are picked up by gdb.

      • sparschauer
        over 6 years ago by sparschauer | Reply

        At least this method works pretty well with the NetIQ GetCore Utility which gathers all loaded ELF files from the crashed system. This way the correct build ids and therefore debuginfo packages are picked up.

    • michalnowak
      over 6 years ago by michalnowak | Reply

      You may find Fedora's ABRT project aligned with your's.

    • sparschauer
      almost 6 years ago by sparschauer | Reply

      @alnovak ignored my merge request for crash-setup. Using the change as custom enhancement if nobody else is interested in it. The core-setup idea has proven working when I picked up the rpms from the supportconfig rpm.txt manually for L3 bug 1105883.

    Similar Projects

    Port git-fixup to POSIX shell script and submit to git/git by mcepl

    Description

    https://github.com/keis/git-fixup is an exceedingly useful program, which I use daily, and I would love to every git user could bask in its awesomeness. Alas, it is a bash script, so it is not appropriate for the inclusion in git proper.

    Goals

    Port the script to plain POSIX shell and submit for consideration to git@vger.kernel.org

    Resources

    FastFileCheck work by pstivanin

    Description

    FastFileCheck is a high-performance, multithreaded file integrity checker for Linux. Designed for speed and efficiency, it utilizes parallel processing and a lightweight database to quickly hash and verify large volumes of files, ensuring their integrity over time.

    https://github.com/paolostivanin/FastFileCheck

    Goals

    • Release v1.0.0

    Design overwiew:

    • Main thread (producer): traverses directories and feeds the queue (one thread is more than enough for most use cases)
    • Dedicated consumer thread: manages queue and distributes work to threadpool
    • Worker threads: compute hashes in parallel

    This separation of concerns is efficient because:

    • Directory traversal is I/O bound and works well in a single thread
    • Queue management is centralized, preventing race conditions
    • Hash computation is CPU-intensive and properly parallelized

    ESETv2 Emulator / interpreter by m.crivellari

    Description

    ESETv2 is an intriguing challenge developed by ESET, available on their website under the "Challenge" menu. The challenge involves an "assembly-like" language and a Python compiler that generates .evm binary files.

    This is an example using one of their samples (it prints N Fibonacci numbers):

    .dataSize 0
.code

loadConst 0, r1 # first
loadConst 1, r2 # second

loadConst 1, r14 # loop helper

consoleRead r3

loop:
    jumpEqual end, r3, r15

    add r1, r2, r4
    mov r2, r1
    mov r4, r2

    consoleWrite r1

    sub r3, r14, r3
    jump loop
end:
hlt

    This language also supports multi-threading. It includes instructions such as createThread to start a new thread, joinThread to wait until a thread completes, and lock/unlock to facilitate synchronization between threads.

    Goals

    • create a full interpreter able to run all the available samples provided by ESET.
    • improve / optimize memory (eg. using bitfields where needed as well as avoid unnecessary memory allocations)

    Resources

    Achivements

    Project still not complete. Added lock / unlock instruction implementation but further debug is needed; there is a bug somewhere. Actually the code it works for almost all the examples in the samples folder. 1 of them is not yet runnable (due to a missing "write" opcode implementation), another will cause the bug to show up; still not investigated, anyhow.

    Add a machine-readable output to dmidecode by jdelvare

    Description

    There have been repeated requests for a machine-friendly dmidecode output over the last decade. During Hack Week 19, 5 years ago, I prepared the code to support alternative output formats, but didn't have the time to go further. Last year, Jiri Hnidek from Red Hat Linux posted a proof-of-concept implementation to add JSON output support. This is a fairly large pull request which needs to be carefully reviewed and tested.

    Goals

    Review Jiri's work and provide constructive feedback. Merge the code if acceptable. Evaluate the costs and benefits of using a library such as json-c.

    FizzBuzz OS by mssola

    Project Description

    FizzBuzz OS (or just fbos) is an idea I've had in order to better grasp the fundamentals of the low level of a RISC-V machine. In practice, I'd like to build a small Operating System kernel that is able to launch three processes: one that simply prints "Fizz", another that prints "Buzz", and the third which prints "FizzBuzz". These processes are unaware of each other and it's up to the kernel to schedule them by using the timer interrupts as given on openSBI (fizz on % 3 seconds, buzz on % 5 seconds, and fizzbuzz on % 15 seconds).

    This kernel provides just one system call, write, which allows any program to pass the string to be written into stdout.

    This project is free software and you can find it here.

    Goal for this Hackweek

    • Better understand the RISC-V SBI interface.
    • Better understand RISC-V in privileged mode.
    • Have fun.

    Resources

    Results

    The project was a resounding success add-emoji Lots of learning, and the initial target was met.