Wiki‎ > ‎

Job submission using multiple nodes and cores

posted Feb 9, 2011, 4:59 PM by Dong Xu   [ updated Jun 9, 2011, 9:54 PM ]
GNU Parallel

http://www.gnu.org/software/parallel/

Youtube intro:

http://www.youtube.com/watch?v=OpaiGYxkSuQ&feature=related
http://www.youtube.com/watch?v=P40akGWJ_gY&feature=related
http://www.youtube.com/watch?v=1ntxT-47VPA&feature=mfu_in_order&list=UL

http://unethicalblogger.com/node/303

http://www.gnu.org/software/parallel/man.html#example__gnu_parallel_as_queue_system_batch_manager

Examples:

ls -d */ |parallel -W /home/lab/usr/dxu/Downloads/dhfr/CPU  --sshloginfile /home/apps/bin/hosts.parallel  'echo $HOSTNAME; cd {}; pwd'

on chili:
nohup sh -c 'ls hc*.namd|parallel --load 0.6 -j 1 "namd2_mc +p48 {} > {}.log"' >& parallel.nohup.log &


https://support.scinet.utoronto.ca/wiki/index.php/User_Serial

Build: https://build.opensuse.org/package/show?package=parallel&project=home%3Atange

GNU Sem: When the number of jobs are running in the background, GNU sem will wait for one of these to complete before starting another command. sem --wait will wait for all jobs to complete.

http://www.gnu.org/software/parallel/man.html#example__working_as_mutex_and_counting_semaphore

Backgroud jobs scripts

http://www.mbeckler.org/scripts.php

http://notepad.cc/nakaro99

xjobs

http://www.maier-komor.de/xjobs.html

http://www.linux.com/archive/feed/57669

http://www.cyberciti.biz/tips/get-more-juice-out-of-multiprocessor-system-with-xjobs.html

http://www.astrobetter.com/multi-core/

xargs -P

http://www.mail-archive.com/bug-findutils@gnu.org/msg02542.html

http://www.xaprb.com/blog/2009/05/01/an-easy-way-to-run-many-tasks-in-parallel/

http://www.mbeckler.org/scripts.php
Comments