I would like to change the way "quilt setup" is implemented.
At the moment, we call rpmbuild and intercept the calls to tar and patch in order to record the location where archives are extracted and the order and options of the patches which apply to them. Then we replay that record to create our own quilt-compatible source tree.
While this works good enough in simple cases, there are two drawbacks:
- We duplicate archive extraction and patch application, which incurs a performance penalty.
- We miss extra commands from the spec file, so the source tree we produce is not exactly what is specified in the spec file.
What I would like to do is keep intercepting calls to patch, but instead of recording them for later use, I'd like to replace them on the fly by quilt import and quilt push. That way rpmbuild will create a quilt-compatible source tree, which we can just copy over to the final location.
There will certainly be a few issue to solve on the way but I can't foresee any blocker. I hope I'm not overseeing a big obvious and unsolvable problem. If we can get there then I expect "quilt setup" to be better and faster.
This project is part of:
Hack Week 11
Activity
Comments
-
almost 10 years ago by jdelvare | Reply
The new backend for "quilt setup" is working, and the performance improvement is very nice. For example, the kernel-default package (which was the slowest one I was aware of) took 1 min 48 s to setup before, while with the new --fast option, it only takes 25 s. So I think the project can be called a success :-)
There are a few technical implementation details which still need to be sorted out, I'll discuss that with upstream. There are some possible improvements left on my to-do list as well.
It took a few tries to get things right. For example, I originally wanted to get rid of the md5sum step altogether, before I remembered that patches are typically passed to us through stdin, so we still need a checksum of all patch files in order to resolve the contents to a filename. I was still able to save time by skipping md5sum on archive files.
Another problem I hit is that my original implementation used "quilt import" to add each patch to the series file. This created a copy of each patch, instead of linking back to the original files in the source directory. That made it impossible to refresh the patches directly as "quilt setup" currently allows. I first worked around that by deleting the copies and linking back afterward, but that was both inefficient and fragile. The proper fix was to manually add the file to the series file instead of calling "quilt import". That required some work so that all the paths were correct both when rpmbuild is applying the patches and later when the working tree is available to the user. The trickiest part was to make it all work also when options -d and/or --sourcedir are used.
Finally, contrary to my original intention, "quilt push" isn't called on every patch. This turned out to be a bad idea from a performance point of view, because individual calls to "quilt push" are much slower than a single call to "quilt push -a". So, it is still up to the user to apply the patches with "quilt push", same as before.
Two user-visible differences exist between the original implementation of "quilt setup" and the new, faster one:
- Headers in the generated series file are incomplete. This prevents reusing the series file for a future call to "quilt setup". I think most users don't care about this feature though, so they won't notice. This could be fixed, but at the price of some of the performance gain, so I don't want to do it by default. Maybe this can be implemented as an option later, but that only really makes sense if we get rid of the original implementation of "quilt setup" (which my patches do not.)
- Patch failures are no longer reported, the user will see them on "quilt push". A great side effect of this is that all patches are added to the series even if one fails to apply in the middle. So I consider this change a feature, as this avoids the "quilt setup" / "rm -rf" cycle the user had to go through before when some patches do not apply.
Also a nice side effect for this project is that I came up with a few bug fixes, cleanups, and performance improvement patches for quilt along the way. 6 of them are already upstream and 6 more have been posted for review.
-
almost 10 years ago by jdelvare | Reply
The result of my work is packaged at: https://build.opensuse.org/package/show/home:jdelvare:branches:devel:tools:scm/quilt
Just pass --fast to "quilt setup" to use the new code, and enjoy the performance boost. If anything doesn't work as expected, please report to me.
Note: for complex packages such as the kernel (or any package including patches in archives), you want to use option -d as well, otherwise some of the performance gain is lost. This is a known bug and I have an idea how it can be fixed, but I did not have the time to implement it yet.
-
almost 10 years ago by jdelvare | Reply
And now as a clickable link: home:jdelvare:branches:devel:tools:scm > quilt
-
-
over 9 years ago by jdelvare | Reply
The quilt setup "fast mode" feature has been committed upstream.
Similar Projects
This project is one of its kind!