Ubuntu: How to use multiple cores with tar gzip compression

 

One thing you may have noticed when using the z switch with tar is that the compression can take some time! If you look at your CPU usage, though, you’ll notice that only one core is being utilised to compress the files. In a modern system 4 or 8 cores are common, meaning that there is plenty of potential to speed up the process if you could utilise more cores. As the gzip package only supports one core, we need to look elsewhere.

 

Fortunately, there is a gzip package which uses multiple cores available – it’s called pigz. To install it type:

 

sudo apt-get install pigz


 

Once that is installed we can tell the tar command to use it like so:

 

tar -c --use-compress-program=pigz  -f [tar file] [directory or files]


 

e.g.:

 

tar -c --use-compress-program=pigz -f backupOfMovies.tar /opt/movies


 

Note the double hyphen before use. Check your CPU usage while the command is running – you should be able to see all available cores being utilised!


Posted

in

, ,

by