Using “parallel” linux command

Parallel is a Linux tool that sends program executions in parallel when multiple processors are available. It works by reading a number of command lines.

As an example let’s use just a command line with echo and sleep

echo "ciao"; sleep 2

We can monitor the time spent using the command time

time echo `echo "ciao"; sleep 2`
ciao
real 0m2.003s
user 0m0.000s
sys 0m0.003s

We can now write twice this command in a file:

cat commands.txt
echo "ciao"; sleep 2
echo "bye"; sleep 2

And then launch it with source:

time source commands.txt
ciao
bye
real 0m4.005s
user 0m0.001s
sys 0m0.003s

We can then use parallel in this way:

time parallel -j 2 < commands.txt
...
ciao
bye
real 0m2.202s
user 0m0.167s
sys 0m0.068s

We use the parameter -j that indicates the maximum number of processes to be used in parallel.

We can also make a for loop that prints the commands to stdout instead of writing a file.

time for ((i=1; i<=5; i++)); do echo "echo $i > $i.tmp; sleep 2"; done | parallel -j 5
real 0m2.293s
user 0m0.178s
sys 0m0.123s

In this way you can replace the echo $i > $i.tmp; sleep 2 with any command-line tool that will be executed in parallel.

Here the same example without parallel

time for ((i=1; i<=5; i++)); do echo $i > $i.tmp; sleep 2; done
real 0m10.070s
user 0m0.001s
sys 0m0.011s

And here an example of using it with 2 processes in parallel with 6 executions.

time for ((i=1; i<=6; i++)); do echo "echo $i > $i.tmp; sleep 2"; done | parallel -j 2
real 0m6.627s
user 0m0.215s
sys 0m0.106s