Using “parallel” linux command
Parallel is a Linux tool that sends program executions in parallel when multiple processors are available. It works by reading a number of command lines.
As an example let’s use just a command line with echo and sleep
echo "ciao"; sleep 2
We can monitor the time spent using the command time
time echo `echo "ciao"; sleep 2`ciaoreal 0m2.003suser 0m0.000ssys 0m0.003s
We can now write twice this command in a file:
cat commands.txtecho "ciao"; sleep 2echo "bye"; sleep 2
And then launch it with source:
time source commands.txtciaobyereal 0m4.005suser 0m0.001ssys 0m0.003s
We can then use parallel in this way:
time parallel -j 2 < commands.txt
...ciaobyereal 0m2.202suser 0m0.167ssys 0m0.068s
We use the parameter -j that indicates the maximum number of processes to be used in parallel.
We can also make a for loop that prints the commands to stdout instead of writing a file.
time for ((i=1; i<=5; i++)); do echo "echo $i > $i.tmp; sleep 2"; done | parallel -j 5real 0m2.293suser 0m0.178ssys 0m0.123s
In this way you can replace the echo $i > $i.tmp; sleep 2 with any command-line tool that will be executed in parallel.
Here the same example without parallel
time for ((i=1; i<=5; i++)); do echo $i > $i.tmp; sleep 2; donereal 0m10.070suser 0m0.001ssys 0m0.011s
And here an example of using it with 2 processes in parallel with 6 executions.
time for ((i=1; i<=6; i++)); do echo "echo $i > $i.tmp; sleep 2"; done | parallel -j 2real 0m6.627suser 0m0.215ssys 0m0.106s