Wiki‎ > ‎

Compile PMEMD on Teragrid

posted Aug 5, 2011, 9:42 PM by Dong Xu
------------ ABE -----------------
Abe uses the Softenv environment to control compilers, mpi implementations
etc. While AMBER can be compiled with any number of combinations of compiler
and MPI library I suggest you use the versions listed below which I have
confirmed to work successfully and with which the benchmarks appearing on
this page were generated.

i) Edit ~/.soft so that it contains the following:

  # Common TeraGrid Software Stack Version 3 (CTSSv3)
  #
  # Please consult TeraGrid user documentation at
  #    http://www.teragrid.org/userinfo/software/softenv.php
  # for information on customizing this file.
  #
  +intel-10.1.017
  +intel-mkl-10.0.3.020
  +mvapich2-1.2-intel-ofed-1.2.
5.5

  # TeraGrid wide basic software suite
  @teragrid-basic

  # TeraGrid wide Globus 4 and Grid software suite
  @globus-4.0

  # Platform recommended development software suite
  @teragrid-dev

Then logout and log back in.

You can check your environment is correctly configured with the following:

  which ifort

This should return: /usr/local/intel/10.1.017/bin/ifort

  which mpif90

This should return: /usr/local/mvapich2-1.2-intel-ofed-1.2.5.5/bin/mpif90

Step 7 - Build PMEMD in parallel
--------------------------------

AMBER 11 features a highly optimized and significantly more scalable MD
engine called PMEMD. This code supports a subset of the sander functionality
but is designed for maximum performance. It is not built by default with the
parallel build above and must be built separately. The following build
instructions can be used to build on Abe:

i) Exit the existing interactive session if it is still running:

  exit

ii) Build PMEMD with the following commands:

  cd $AMBERHOME/src/pmemd/
  ./configure linux_em64t ifort mvapich pubfft bintraj

When prompted for the location of the infiniband libraries enter:

   /usr/local/mvapich2-1.2-intel-ofed-1.2.5.5/lib/

We will now edit the config.h file to optimize the compilation flags for the
Clovertown processors on Abe. Edit config.h and change the following lines:

  MPI_LIBS = -L$(MPI_LIBDIR) -lmpich -L$(MPI_LIBDIR2) -lmtl_common -lvapi
-lmosal -lmpga -lpthread

  F90 = ifort

  F90_OPT_LO =  -tpp7 -O0
  F90_OPT_MED = -tpp7 -O2
  F90_OPT_HI =  -tpp7 -xP -ip -O3

  LOAD = ifort

to

  MPI_LIBS =

  F90 = mpif90

  F90_OPT_LO =  -O0
  F90_OPT_MED = -O2
  F90_OPT_HI =  -fast -axPTS -O3

  LOAD = mpif90

Now we can build it with:

  make
  make install

This will create the executable pmemd in $AMBERHOME/exe/

Step 8 - Test PMEMD in parallel
------------------------------

The final stage is to test PMEMD in parallel. As before we will use an
interactive shell to do this:

i) Request an interactive session of 4 processors on 1 node with the
command:

  qsub -I -V -l walltime=02:00:00,nodes=1:ppn=4

ii) Once the job starts and you have a shell command line you can run the
test cases in parallel as follows:

  setenv AMBERHOME ~/amber10
  cd $AMBERHOME/test
  setenv DO_PARALLEL 'mpirun -np 4 -machinefile $PBS_NODEFILE '
  mvapich2-start-mpd
  make clean
  make test.pmemd
  mpdallexit

Again check for TEST_FAILURES.diff and see if any tests failed.

Compilation is now complete and AMBER is ready for production runs.

Example Job Submission Script
-----------------------------

The following is an example job submission script for running PMEMD on
parallel on Abe.

#!/bin/tcsh
#PBS -l walltime=24:00:00
# Run on 64 cores
#PBS -l nodes=8:ppn=8
# export all my environment variables to the job
#PBS -V
# job name (default = name of script file)
#PBS -N testjob
#PBS -A account_number_to_charge

mvapich2-start-mpd
setenv NP `wc -l ${PBS_NODEFILE} | cut -d'/' -f1`

setenv AMBERHOME ~/amber10
setenv MV2_SRQ_SIZE 4000

cd /scratch/directory_to_run_job_in/

mpirun -machinefile ${PBS_NODEFILE} -np ${NP} $AMBERHOME/exe/pmemd -O \
    -i mdin -o mdout \
    -p prmtop -c inpcrd \
    -r restrt -x mdcrd </dev/null

mpdallexit

-----------------------------


------------------ RANGER ------------------

Step 1 - Prepare your environment
---------------------------------

Ranger uses a module system for controlling your compilers, mpi
implementations etc. While AMBER can be compiled with any number of
combinations of compiler and MPI library I suggest you use the versions
listed below which I have confirmed to work successfully and with which the
benchmarks appearing on this page were generated.

i) You should make sure the following modules are loaded by default for your
  account:

  pgi/7.2-5
  mvapich/1.0.1

  These are, at the time of writing, the current defaults on Ranger so you
should not need to change anything. You can check the currently loaded
modules with:

  module list

  This will provide something similar to the following:

Currently Loaded Modules:
 1) TACC-paths     6) binutils-amd/070220  11) tgproxy/0.9.1  16)
TERAGRID-BASIC  21) gzip/1.3.12
 2) Linux          7) TERAGRID-paths       12) tgresid/2.0.3  17)
globus/4.0.8    22) tar/1.22
 3) cluster-paths  8) gx-map/0.5.3.3       13) tgusage/3.0    18)
GLOBUS-4.0      23) cluster
 4) pgi/7.2-5      9) srb-client/3.4.1     14) uberftp/2.4    19)
TERAGRID-DEV    24) TACC
 5) mvapich/1.0.1  10) tg-policy/0.2       15) tginfo/1.0.1   20) CTSSV4
25) mkl/10.0

  You should also check the correct mvapich installation is being used. Run
the following

  which mpif90

  This should return

  /opt/apps/pgi7_2/mvapich/1.0.1/bin/mpif90

  If it does not then you will need to unload whichever module is defining
the mpi that is reported to be in use and load the one defined above.

Step 7 - Build PMEMD in parallel
--------------------------------

AMBER 11 features a highly optimized and significantly more scalable MD
engine called PMEMD. This code supports a subset of the sander functionality
but is designed for maximum performance. It is not built by default with the
parallel build above and must be built separately. The following build
instructions can be used to build on Ranger:


i) Build PMEMD with the following commands:

  cd $AMBERHOME/src/pmemd/
  ./configure linux64_opteron pgf90 mvapich pubfft bintraj

When prompted for the location of the infiniband libraries enter:

   /opt/apps/pgi7_2/mvapich/1.0.1/

We will now edit the config.h file to optimize the compilation flags for the
AMD Barcelona chips on Ranger as well as making sure it links the mpi
libraries correctly. Edit config.h and change the following lines:

  MPI_LIBS = -L$(MPI_LIBDIR) -lmpich -L$(MPI_LIBDIR2) -lmtl_common -lvapi
-lmosal -lmpga -lpthread

  F90 = pgf90

  LOAD = ifort

to

  MPI_LIBS =

  F90 = mpif90

  LOAD = mpif90

Now we can build it with:

  make
  make install

This will create the executable pmemd in $AMBERHOME/exe/

Step 8 - Test PMEMD in parallel
------------------------------

The final stage is to test PMEMD in parallel. As before we will submit a job
to the queue that will run the tests for us.

i) Clean the test directory with:

  setenv AMBERHOME ~/amber10
  cd $AMBERHOME/test
  make clean

ii) Create the following job submission script.


 #!/bin/csh
 #$ -V
 #$ -A YOUR_ACCOUNT_NUMBER
 #$ -N test_sander.MPI
 #$ -j y
 #$ -M YOUR@EMAIL.ADDRESS
 #$ -l h_rt=01:00:00
 #$ -o $JOB_NAME.o$JOB_ID
 #$ -pe 4way 16            # Requests 4 cores/node, 4/16 = 1 node total
 #$ -q development

 setenv AMBERHOME ~/amber10/
 cd $AMBERHOME/test/
 date

 setenv DO_PARALLEL ibrun
 make test.pmemd

Again check for TEST_FAILURES.diff and see if any tests failed.

Compilation is now complete and AMBER is ready for production runs.

Example Job Submission Script
-----------------------------

The following is an example job submission script for running PMEMD on
parallel on Ranger.

 #!/bin/csh
 #$ -V
 #$ -A YOUR_ACCOUNT_NUMBER
 #$ -N test_sander.MPI
 #$ -j y
 #$ -M YOUR@EMAIL.ADDRESS
 #$ -l h_rt=24:00:00
 #$ -o $JOB_NAME.o$JOB_ID
 #$ -pe 4way 32            # Requests 16 cores/node, 16/32 = 2 nodes total
 #$ -q normal

 setenv AMBERHOME ~/amber10/
 cd $AMBERHOME/test/
 date

 ibrun $AMBERHOME/exe/pmemd -O -i mdin -o mdout -p prmtop -c inpcrd -x
mdcrd </dev/null
===============================================
When compiling PMEMD, Add -DNO_NTT3_SYNC to the config.h file - on the same line as -DMPI etc. This turns off random number synching for ntt=3 and makes performance MUCH better
in parallel. It will though make the ntt=3 test cases fail but this is
benign.

The standard way langevin dynamics is run is to use a single random number
generator and sync it across threads. This means you get the same random
number stream across multiple numbers of processors and thus reproducibility
of results. Of course this leads to scaling issues as you go to larger core
counts since cycling through the random numbers and chucking them away on
each core is a serial bottleneck and slows things down considerably.

One can add -DNO_NTT3_SYNC to avoid this synchronization when using ntt=3.
This improves scaling for Langevin runs but now the random number streams
are different on different threads. Hence there is no longer any
reproducibility across different numbers of MPI threads. Of course the
answers are no less wrong, just the test case outputs do not match. Hence
the reason it is an 'undocumented compile time feature'. Ultimately we need
to add a true parallel random number generator but for the moment this
suffices.
Comments