GROMACS (www.gromacs.org) is a versatile package to perform molecular dynamics, i.e. simulate the Newtonian equations of motion for systems with hundreds to millions of particles. It is primarily designed for biochemical molecules like proteins and lipids that have a lot of complicated bonded interactions, but since GROMACS is extremely fast at calculating the nonbonded interactions (that usually dominate simulations) many groups are also using it for research on non-biological systems, e.g. polymers.

GROMACS manual, downloadable in several formats.

Versions

For basic information about setting up GROMACS jobs, read the GROMACS documentation. A collection of sample jobs is in /usr/local/gromacs/share/tutor.

Biowulf nodes have 2 (default gige nodes, ipath nodes), 4 (dual-core 'dc' nodes) or 8 (IB nodes) processors. The number of processors should be appropriately chosen depending on the type of node.

Sample script for a GROMACS 4.x run on Infiniband:

#!/bin/bash
# this file is Run_Gromacs
#PBS -N Gromacs
#PBS -k oe
#PBS -m be

# set up PATH for Infiniband nodes
export PATH=/usr/local/openmpi_ib/bin:/usr/local/gromacs/bin:$PATH

cd /data/user/my_gromacs_dir

grompp > outfile 2>&1
`which mpirun` -machinefile $PBS_NODEFILE -n $np `which mdrun_mpi` >> outfile 2>&1

To run GROMACS on Infinipath or Ethernet only the PATH to mpirun needs to change.
For an Infinipath run:

# set up PATH for Infinipath nodes
export PATH=/usr/local/openmpi_ipath/bin:/usr/local/gromacs/bin:$PATH

For GROMACS over Ethernet

# set up PATH for Ethernet nodes
export PATH=/usr/local/openmpi_eth/bin:/usr/local/gromacs/bin:$PATH

The script can be submitted with the qsub command. The number of processes should be chosen to match the number of cores on the node. The cores on each type of node is listed in the second column of the 'freen' command.

Submitting to IB nodes (8 processors per node):

qsub -v np=32 -l nodes=4:ib Run_Gromacs

Submitting to Ipath nodes (2 processors per node):

qsub -v np=16 -l nodes=8:ipath Run_Gromacs 

Submitting to gige nodes (2 processors per node):

qsub -v np=4 -l nodes=2 Run_Gromacs

Submitting to dual-core gige nodes (4 processors per node):

qsub -v np=8 -l nodes=2:dc Run_Gromacs

Gromacs 4.5.5 has also been built with Plumed 1.3, a plugin for free energy calculations in molecular systems. Plumed website. Free energy calculations can be performed as a function of many order parameters with a particular focus on biological problems, using state of the art methods such as metadynamics, umbrella sampling and Jarzynski-equation based steered MD.

This version of Gromacs is in /usr/local/GROMACS/4.5.5+plumed. Sample batch script:

#!/bin/bash

#PBS -N myjob
#PBS -m be

export PATH=/usr/local/openmpi_gnu_eth/bin:/usr/local/GROMACS/4.5.5+plumed/bin:$PATH
`which mpirun` -machinefile $PBS_NODEFILE -n $np `which mdrun_mpi` >> outfile 2>&1

To run on Ipath nodes, use the path

export PATH=/usr/local/openmpi_ipath/bin:/usr/local/GROMACS/4.5.5+plumed/bin:$PATH

To run on IB nodes, use the path

export PATH=/usr/local/openmpi_ib/bin:/usr/local/GROMACS/4.5.5+plumed/bin:$PATH

Details about running replica exchange with Gromacs are on the Gromacs website. Multiple tpr files need to be generated from multiple *.mdp files with different temperatures. Below is a sample script for generating the tpr files. (courtesy Jeetain Mittal, NIDDK)

#!/bin/csh -f 

set ff = $argv[1]
set s = $argv[2]
set proot = 2f4k
#set ff = amber03d
set i = 0

while ( $i < 40 ) 

set fileroot = "${proot}_${ff}"
set this = "trexr"

if ( $s == 1 ) then 
set mdp = "mdp/trex_ini${i}.mdp"
set gro = "unfolded.gro" 
else
set sprev = $s
@ sprev--
set mdp = "mdp/trex_cont${i}.mdp"
set gro = "data/gro/${fileroot}_${this}_s${sprev}_nd${i}.gro"
endif

# 40 rep
grompp -v -f $mdp -c $gro \
-o data/tpr/${fileroot}_${this}_nd${i}.tpr \
-p ${fileroot}_ions.top

@ i++
end

Gromacs 4.0 can run with each replica on multiple processors. It is most efficient to run each replica on a dual-core node using all the processors on that node. This requires creating a specialized list of processors with the command make-gromacs-nodefile-dc (which is in /usr/local/bin) as in the sample script below.

#!/bin/bash
# this file is Run_Gromacs_RE
#PBS -N Gromacs_RE
#PBS -k oe
#PBS -m be

# set up PATH for gige or ib nodes
export PATH=/usr/local/openmpi/bin:/usr/local/gromacs/bin:$PATH

cd /data/user/my_gromacs_dir

#create the specialized list of processors for RE
make-gromacs-nodefile-dc

/usr/local/openmpi/bin/mpirun -machinefile ~/gromacs_nodefile.$PBS_JOBID \
      -np $np /usr/local/gromacs/bin/mdrun_mpi \ 
      -multi $n -replex 2000 >> outfile 2>&1

Submit this script to the dual-core nodes with:

qsub -v np=128,n=32 -l nodes=32:dc Run_Gromacs_RE

The above command will submit the job to 32 dual-core (either o2800 or o2600) nodes. Each of the 32 replicas will run on all 4 processors of each node. The number of processors (np=128) and the number of nodes (n=32) is passed to the program via the -v flag in qsub.

It is critical to determine the appropriate number of nodes on which to run your job. As shown in the benchmarks below, different jobs scale differently. Thus, one job which scales very well could be submitted on up to 10 nodes, while another job may scale only up to 2 nodes. For some jobs, if you submit to more nodes than is optimal, your job will actually run slower.

To determine the optimal number of nodes:

• Set up a small version of your job. e.g. if your project involves a simulation of 100 ps, set up a 1ps job.
• Submit this job to 2 processors (1 node), 4processors (2 nodes), 6 processors (3 nodes) ...
• Examine the results. You want to pick the number of processors with at least 70% efficiency.

  
  100 * Time on 1 processor
  ---------------------------    = Efficiency  
  n * Time on n processors
  

• Biowulf job monitors. Click on 'List status of Running Jobs only', and then on your username in the resultant web page. This will pop up a small window showing the status of all nodes that are running your jobs, as in the picture on the right. In the ideal situation, all your nodes should be yellow (both processors used). Watch for green or red nodes. Clicking on a node will pop up a window with information about the processes running on that node.

• jobload. The jobload command will report the loads on the nodes allocated to your jobs.

  biowulf% jobload userA
  Jobs for  userA      Node    Load    
  1708520.biobos      p289     75%
                      p293     74%
                      p297     90%
                      p298     67%
              Job Average:     76%
  1952867.biobos      p308     71%
                      p363     70%
                      p369     69%
                      p377     86%
              Job Average:     74%
  
  User Average:       75%
  

• The PBS batch system will write 2 files called JOBNAME.oJOBNUM and JOBNAME.eJOBNUM in your home directory (e.g. GROMACS.o90763 and GROMACS.e90763). They contain the standard output and standard error from your job. Most problems can be figured out by examining these files.

Summary:

• Different Gromacs jobs scale differently; it is important to run your own benchmarks to determine the optimal number of nodes for your particular job.
• Gromacs scales best on the Infiniband nodes. Based on the d.dppc benchmark results below, one could submit this job to 128 IB or Ipath processors, or 16 gige processors.

Benchmarks for Gromacs 4.0.3

Benchmarks for Gromacs 3.3.3.

Benchmarks for Gromacs 3.3.1