Looking for help improving performance of GNU Make
I've had luck using https://github.com/dmoulding/boilermake
It's a GNU Make framework which allows you to create non-recursive Makefiles. I've re-worked it a bit, and used it in my pet project: https://github.com/freeradius/freeradius-server/
100K+ LoC, ~400 source files, multiple subdirectories, building libraries, executables, etc.
Most of the slowness of Make is due to people creating recursive Makefiles. Running endless shell programs is very slow. Make by itself is surprisingly fast.
The other benefit of non-recursive Make is that the builds can now go in parallel. What used to be a ~90s build is now ~7s on a multi-core machine.
I also replace libtool (shudder) with jlibtool. That was taken from Apache, and modified to fix bugs work better, etc. That sped up the build a lot, too.
Maybe I'm crazy, but these tools have worked across all Unix-y platforms I've tried. And I needed a migration path from recursive Make / libtool to something better. Dropping the build system on the floor and hoping for something better wasn't an option. This tool suite works much like the old one, but it's simpler, faster, and doesn't require additional packages (python, etc.) to do the build. Everything outside of Make is self-contained, and is included in the FreeRADIUS distribution.
My projects:
https://github.com/alandekok/boilermake
I really hope someone takes the job offer, fixes the problem however possible, and then publicises the process.
This is what free software is supposed to be about: you can hire anyone to fix it. Let's put theory to practice now and actually hire someone to do it.
Edit: You know what, I'm now determined to now headhunt this person. It's the principle of the thing. Once I get the original poster's permission, I am reposting this ad on freedomsponsors.org, bountysource.com, and on the GNU job list: http://www.fsf.org/resources/jobs
Summary:
> What we see is that if we have more than 30 or so clients connected to the build Make is often unable to generate tasks fast enough to keep them busy.
[…]
> Our management doesn't want us taking time to learn the internals of make and see if we can improve it ourselves, but they are willing to throw money at the problem, either by offering some sort of donation/bounty for performance improvements or directly hiring an expert on a contract basis; any improvements achieved would of course be submitted back to the community as patches.
Last month at EuroBSDCon, Marc Espie gave a neat talk describing some of the performance improvements he’s made to OpenBSD’s Make (and some positive words about Ninja as well) – http://www.openbsd.org/papers/eurobsdcon2014-parallel-make.p...
Make has so much legacy cruft handling, people just moved to other systems that are simpler.
Most developers don't even know most of the features make, and people don't really want to learn such an old program through and through.
If they're building C/C++ with it then this is a good option: https://github.com/Zomojo/Cake - which is really just a front end to make, but produces really fast makefiles.
It was initially developed to allow interactive speed iteration of compiling large C/C++ projects for data analysis.
It does require that your source code comply with a few basic rules (just that c/cpp files and headers have the same name). But beyond that it runs automatically and quickly.
undefined
Unfixable. Use ninja instead.