Speedup GNU make build and compilation process
Building/compiling libraries or projects frequently with GNU make takes long time. To speed up these builds you can use make -jN where N is the number of parallel builds. For example:
make -j4
will make 4 parallel builds.
Hardware considerations
Of course, you can’t do something like
make -j256
This is the best way to get your computer hangs !!! You must check your hardware CPU informations. How many processors (cores) you have (including multi-threading):
NB_CORES=$(grep -c '^processor' /proc/cpuinfo)
You can also use dmidecode :
dmidecode -t processor | grep "Core Count"
Core Count: 8
Core Count: 8
here you got two processors (2 times Core Count) with 8 cores. You can also use lspcu command:
lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 4
On-line CPU(s) list: 0-3
Thread(s) per core: 1
Core(s) per socket: 4
Socket(s): 1
NUMA node(s): 1
Vendor ID: GenuineIntel
CPU family: 6
Model: 23
Stepping: 7
CPU MHz: 1998.000
BogoMIPS: 4999.98
Virtualization: VT-x
L1d cache: 32K
L1i cache: 32K
L2 cache: 3072K
NUMA node0 CPU(s): 0-3
System considerations
Once you get your number of cores, we must keep a responsive system. We use -l options of make -l [load], –load-average[=load] {Specifies that no new jobs (commands) should be started if there are others jobs running and the load average is at least} load
To get a full saturation of the processors without hanging your system, export the following environment variable MAKEFLAGS:
NB_CORES=$(grep -c '^processor' /proc/cpuinfo)
export MAKEFLAGS="-j$((NB_CORES+1)) -l${NB_CORES}"
make
Or:
NB_CORES=$(grep -c '^processor' /proc/cpuinfo)
make -j$((NB_CORES+1)) -l${NB_CORES}
It means here that the load average does not exceed number of cores.
Persistent settings
To get these settings persistent, just add in your .bashrc or .bash_profile:
NB_CORES=$(grep -c '^processor' /proc/cpuinfo)
export MAKEFLAGS="-j$((NB_CORES+1)) -l${NB_CORES}"
At last a commandline example
I you dont want particular settings , just use commandline:
make -j9 -l8
Remarks
You can usually get away with having twice as many jobs as you do CPUs if you have a decent amount of RAM. If your CPU supports hyper-threading then you can often get by with twice as many again. That’s 128 jobs if you have a 32 dual core machine with hyper-threading! (Thanks to Andrew Stormont for his contribution)
If you found this post or this website helpful and would like to support our work, please consider making a donation. Thank you!
Help UsArticles in the same category
- Time a task: time
- SVN -- How to ignore file or directory in subversion?
- SVN -- Branch, Branching subversion howto
- Speedup GNU make build and compilation process
- phpMyAdmin: Search and Replace in MySQL database
- Linux How to delete or remove printer from command line
- Linux How to connect to Windows with remote desktop RDP in CentOS 7 / RedHat 7
- Intel compilation for MIC architecture KNL Knights Landing
- How to setup SSH timeout in shell script ?
- How to make a denial of a service with fork functions in BASH ?
- How to encrypt/decrypt a file or directory in Linux?
- How to diff remote files using ssh ?
- How to Convert Text File From ISO-8859-15 to UTF-8 Encoding
- How to change the MAC address on Linux
- Got permission denied while trying to connect to the Docker daemon socket
- GNU compilation for MIC architecture KNL Knights Landing
- Generating a self-signed certificate using OpenSSL with Linux CentOs/RedHat for Apache/httpd
- Find out biggest cpu/memory consuming processes with ps command
- Find list of options that python was compiled with
- Download music and videos .mp3, .wma, .avi, .mpg , divx with google
- Crontab : Scheduling Tasks
- Comment changer son adresse MAC sous Linux
- Check/find version of numpy i'm using
- Archiving and compressing data files tar
- Linux - Tip of the day