Math-Linux.com

Knowledge base dedicated to Linux and applied mathematics.

Home > Linux > Tip of the day > Speedup GNU make build and compilation process

Speedup GNU make build and compilation process

All the versions of this article: <English> <français>

Building/compiling libraries or projects frequently with GNU make takes long time. To speed up these builds you can use make -jN where N is the number of parallel builds. For example:


make -j4

will make 4 parallel builds.

Hardware considerations

Of course, you can’t do something like


make -j256

This is the best way to get your computer hangs !!!
You must check your hardware CPU informations. How many processors (cores) you have (including multi-threading):


NB_CORES=$(grep -c '^processor' /proc/cpuinfo)

You can also use dmidecode :


dmidecode -t processor | grep "Core Count"
Core Count: 8
Core Count: 8

here you got two processors (2 times Core Count) with 8 cores. You can also use
lspcu command:


lscpu
Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                4
On-line CPU(s) list:   0-3
Thread(s) per core:    1
Core(s) per socket:    4
Socket(s):             1
NUMA node(s):          1
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 23
Stepping:              7
CPU MHz:               1998.000
BogoMIPS:              4999.98
Virtualization:        VT-x
L1d cache:             32K
L1i cache:             32K
L2 cache:              3072K
NUMA node0 CPU(s):     0-3

System considerations

Once you get your number of cores, we must keep a responsive system. We use -l options of make
-l [load], —load-average[=load]
Specifies that no new jobs (commands) should be started if there are others jobs running and the load average is at least load

To get a full saturation of the processors without hanging your system, export
the following environment variable MAKEFLAGS:


NB_CORES=$(grep -c '^processor' /proc/cpuinfo)
export MAKEFLAGS="-j$((NB_CORES+1)) -l${NB_CORES}"
make

Or:


NB_CORES=$(grep -c '^processor' /proc/cpuinfo)
make -j$((NB_CORES+1)) -l${NB_CORES}

It means here that the load average does not exceed number of cores.

Persistent settings

To get these settings persistent, just add in your .bashrc or .bash_profile:


NB_CORES=$(grep -c '^processor' /proc/cpuinfo)
export MAKEFLAGS="-j$((NB_CORES+1)) -l${NB_CORES}"

At last a commandline example

I you dont want particular settings , just use commandline:


make -j9 -l8

Remarks

You can usually get away with having twice as many jobs as you do CPUs if you have a decent amount of RAM. If your CPU supports hyper-threading then you can often get by with twice as many again. That’s 128 jobs if you have a 32 dual core machine with hyper-threading! (Thanks to Andrew Stormont for his contribution)

Also in this section

  1. Crontab : Scheduling Tasks
  2. Archiving and compressing data files tar
  3. Check/find version of numpy i’m using
  4. Comment changer son adresse MAC sous Linux
  5. Download music and videos .mp3, .wma, .avi, .mpg , divx with google
  6. Find list of options that python was compiled with
  7. Find out biggest cpu/memory consuming processes with ps command
  8. Generating a self-signed certificate using OpenSSL with Linux CentOs/RedHat for Apache/httpd
  9. GNU compilation for MIC architecture KNL Knights Landing
  10. Got permission denied while trying to connect to the Docker daemon socket
  11. How to change the MAC address on Linux
  12. How to Convert Text File From ISO-8859-15 to UTF-8 Encoding
  13. How to diff remote files using ssh ?
  14. How to encrypt/decrypt a file or directory in Linux?
  15. How to make a denial of a service with fork functions in BASH ?
  16. How to setup SSH timeout in shell script ?
  17. Intel compilation for MIC architecture KNL Knights Landing
  18. Linux How to connect to Windows with remote desktop RDP in CentOS 7 / RedHat 7
  19. Linux How to delete or remove printer from command line
  20. phpMyAdmin: Search and Replace in MySQL database
  21. Speedup GNU make build and compilation process
  22. SVN — Branch, Branching subversion howto
  23. SVN — How to ignore file or directory in subversion?
  24. Time a task: time