New Development
In previous hack weeks, the first few days ended up being wasted on just getting it working. I'm pleased to share that the code quality has improved dramatically since the last hack week and there are now extensive test cases for both unit testing and testing against real vmcores, and we'll use both mypy and pylint (if installed) to perform static analysis. Packages for those are available in openSUSE or as part of the crash-python OBS repo for SLE15. It has been tested with kernels from 3.0 to 5.1.
I have also written some documentation. It's by no means complete but it should be easier to dig in than in previous Hack Weeks.
This Hack Week, I'd like to focus on implementing the commands that are still not present that users of legacy crash are used to seeing.
Major Changes
Recent development has changed runtime requirements and internal APIs substantially. Previous uncommitted work will likely be incompatible. I think you'll find the changes worthwhile, though.
Where to start
Documentation is available on readthedocs.io.
Updated tools from the home:jeff_mahoney:crash-python OBS project are required and older versions are incompatible. Packages for libkdumpfile, pyelftools, crash-python-gdb, and crash-python itself are all there.
The latest crash-python code is available on GitHub, using the master branch.
There is still an EtherPad but it hasn't seen much activity.
Historical Background
Crash, the Linux kernel core dump debugger, has the ability to use extensions written in C, but oftentimes we want to do some automatic analysis or a dump of data that is more involved than what Crash offers natively. Since GDB 7.0, GDB has incorporated an embedded python interpreter.
Since we started hacking on this project during Hack Week several years ago, we've made significant progress in making crash-python more usable. The initial plan of exporting crash functionality via Python was ultimately discarded in favor of writing a kernel debugger in Python with as few GDB extensions as possible. The GDB extensions have long since been written and are entirely generic. We export some ELF functionality as well as add the ability to implement a GDB Target in Python.
Since early 2016, the target has been implemented and interfaces with Petr Tesarik's libkdumpfile python module. The crash-python project uses that to offer some basic kernel debugging functionality: We can do 'dmesg', 'sys', and subsets of 'ps' and 'kmem.' We can also load up the tasks from the crash dump as GDB threads and do full symbolic debugging (with the caveat that the kernel is built with -O2 and some symbols are optimized away.) Some colleagues have used it to hunt real bugs in the course of their normal work!
Looking for hackers with the skills:
Nothing? Add some keywords!
This project is part of:
Hack Week 11 Hack Week 12 Hack Week 13 Hack Week 14 Hack Week 15 Hack Week 16 Hack Week 17 Hack Week 18 Hack Week 19 Hack Week 20 Hack Week 21
Activity
Comments
-
about 10 years ago by ptesarik | Reply
The main difficulty is that crash translates everything to PHYSICAL addresses, whereas gdb uses VIRTUAL addresses only. Kernel crash dumps generally do not contain all existing virtual mappings of physical RAM, so gdb does not find the data. I'm not sure how difficult it would be to implement a kind of virttophys() in gdb, but we're pretty much lost without that.
-
-
about 10 years ago by jeff_mahoney | Reply
Useful links:
https://build.opensuse.org/project/show/home:jeff_mahoney:py-crash https://etherpad.nue.suse.com/p/crash-todo https://github.com/jeffmahoney/py-crash
-
about 10 years ago by jeff_mahoney | Reply
@Petr For the python code I've already implemented (see github link), it does the translation already.
-
about 10 years ago by ptesarik | Reply
Yes, but most of it is missing. It does not handle relocated kernels, vmemmap, kernel modules and a lot of other stuff.
-
about 10 years ago by jeff_mahoney | Reply
Yep. It wasn't intended to be complete yet. It's more of a starting point.
-
-
about 10 years ago by ptesarik | Reply
And most importantly, the interface to gdb still uses virtual addresses. Note that the dump file may not contain the identity-mapped page of all physical pages; and some architectures do not even map all physical pages to linear address space (yeah, I know that PAE sucks, but...).
-
Similar Projects
This project is one of its kind!