Math-Linux.com

Knowledge base dedicated to Linux and applied mathematics.

Home > Linux > Tip of the day > Speedup GNU make build and compilation process

Speedup GNU make build and compilation process

All the versions of this article: <English> <français>


Building/compiling libraries or projects frequently with GNU make takes long time. To speed up these builds you can use make -jN where N is the number of parallel builds. For example:

make -j4

will make 4 parallel builds.

Hardware considerations

Of course, you can’t do something like

make -j256

This is the best way to get your computer hangs !!!
You must check your hardware CPU informations. How many processors (cores) you have (including multi-threading):

NB_CORES=$(grep -c '^processor' /proc/cpuinfo)

You can also use dmidecode :

dmidecode -t processor | grep "Core Count"
Core Count: 8
Core Count: 8

here you got two processors (2 times Core Count) with 8 cores. You can also use
lspcu command:

lscpu
Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                4
On-line CPU(s) list:   0-3
Thread(s) per core:    1
Core(s) per socket:    4
Socket(s):             1
NUMA node(s):          1
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 23
Stepping:              7
CPU MHz:               1998.000
BogoMIPS:              4999.98
Virtualization:        VT-x
L1d cache:             32K
L1i cache:             32K
L2 cache:              3072K
NUMA node0 CPU(s):     0-3

System considerations

Once you get your number of cores, we must keep a responsive system. We use -l options of make
-l [load], —load-average[=load]
Specifies that no new jobs (commands) should be started if there are others jobs running and the load average is at least load

To get a full saturation of the processors without hanging your system, export
the following environment variable MAKEFLAGS:

NB_CORES=$(grep -c '^processor' /proc/cpuinfo)
export MAKEFLAGS="-j$((NB_CORES+1)) -l${NB_CORES}"
make

Or:

NB_CORES=$(grep -c '^processor' /proc/cpuinfo)
make -j$((NB_CORES+1)) -l${NB_CORES}

It means here that the load average does not exceed number of cores.

Persistent settings

To get these settings persistent, just add in your .bashrc or .bash_profile:

NB_CORES=$(grep -c '^processor' /proc/cpuinfo)
export MAKEFLAGS="-j$((NB_CORES+1)) -l${NB_CORES}"

At last a commandline example

I you dont want particular settings , just use commandline:

make -j9 -l8

Remarks

You can usually get away with having twice as many jobs as you do CPUs if you have a decent amount of RAM. If your CPU supports hyper-threading then you can often get by with twice as many again. That’s 128 jobs if you have a 32 dual core machine with hyper-threading! (Thanks to Andrew Stormont for his contribution)

Also in this section

  1. Crontab : Scheduling Tasks
  2. Time a task: time
  3. Download music and videos .mp3, .wma, .avi, .mpg , divx with google
  4. How to change the MAC address on Linux
  5. Comment changer son adresse MAC sous Linux
  6. How to Convert Text File From ISO-8859-15 to UTF-8 Encoding
  7. SVN — How to ignore file or directory in subversion?
  8. How to setup SSH timeout in shell script ?
  9. phpMyAdmin: Search and Replace in MySQL database
  10. How to encrypt/decrypt a file or directory in Linux?
  11. Find list of options that python was compiled with
  12. Check/find version of numpy i’m using
  13. Speedup GNU make build and compilation process
  14. How to make a denial of a service with fork functions in BASH ?
  15. Intel compilation for MIC architecture KNL Knights Landing
  16. GNU compilation for MIC architecture KNL Knights Landing
  17. Archiving and compressing data files tar
  18. Linux How to delete or remove printer from command line
  19. Linux How to connect to Windows with remote desktop RDP in CentOS 7 / RedHat 7
  20. SVN — Branch, Branching subversion howto
  21. Generating a self-signed certificate using OpenSSL with Linux CentOs/RedHat for Apache/httpd
  22. Got permission denied while trying to connect to the Docker daemon socket
  23. Find out biggest cpu/memory consuming processes with ps command
  24. How to diff remote files using ssh ?