From the getting your cores off the sofa department

Every now and then there are really nifty multi-core applications that help with some of the more mundane Linux HPC chores. The -j option for make is one such example. I recently stumbled upon two other applications that take advantage of multi-core for file compression.

The two packages are parallel gzip and parallel bzip2. As you might surmise, each application uses multiple cores to speed up the compression of large files. I checked to see if these were in the Scientific Linux Yum repository, but they did not show up when I tried to install them. Thus, I decided to build them myself.

Downloading an building was simple (make sure you have bzip2-devel and zlib-devel libraries installed) I actually created a spec files for them and produced my own RPMs to be used in my collection of cluster packages.

Of course, building packages is only half the fun. I decided to test the packages on my Limulus personal cluster. The head node is a currently a quad-core Intel Q6600 running Scientific Linux 6.0. In order to create a large file, I used tar and archived /usr/share (assuming it had a mix of file types):

$ tar cvf usr.share.tar /usr/share

The resulting file was 1.1GB, which was big enough for my simple tests. I then tried both the serial and the parallel versions on the file, both compressing and decompressing. Note, decompression is a mostly a serial task, thus does not benefit from multiple cores. To be a bit more formal, I created a simple script (see below) that all runs the tests and check to see if the file survived. I then placed the results in the following table.

Table One: Results for various parallel compression packages for a 1.1GB file

The results were impressive. Basically I saw a ~3.5 times speed up for either method. The sequential bzip2 compression takes much longer than the sequential gzip, but the compression can be much better. In this case, the extra compression was not all that great, but in cases were the file is more "compressible," bzip2 will create much smaller files. In the case of pbzip2, it can make the compression times for bzip2 more tolerable. Overall, a nice pay-off for less than an hour of compiling and testing. Now I'll get back to my regular work, which quite interestingly involves creating some disk images with dd. Now all I need is fast compression tool ...

  #!/bin/bash

  #Create file
  tar cvf usr.share.tar /usr/share/  

  #Initial File size
  echo "Initial size and md5"
  md5sum usr.share.tar
  ls -lh usr.share.tar

  #gzip
  echo "Sequential gzip"
  time gzip usr.share.tar
  ls -lh usr.share.tar.gz
  gunzip usr.share.tar.gz

  #pigz
  echo "Parallel pigz"
  time pigz usr.share.tar
  ls -lh usr.share.tar.gz
  unpigz usr.share.tar.gz

  #bzip2
  echo "Sequential bzip2"
  time bzip2 usr.share.tar
  ls -lh usr.share.tar.bz2
  bunzip2 usr.share.tar.bz2

  #pbzip
  echo "Parallel pbzip2"
  time pbzip2 usr.share.tar
  ls -lh usr.share.tar.bz2
  bunzip2 usr.share.tar.bz2

  # Final File size
  echo "Final size and md5"
  md5sum usr.share.tar
  ls -lh usr.share.tar

You have no rights to post comments