How to Run GROMACS on an HPC Cluster with SLURM: Job Scripts, Modules and Restarts

To run GROMACS on an HPC cluster you load the site’s GROMACS module, then submit an sbatch script that requests cores and GPUs, exports OMP_NUM_THREADS from SLURM_CPUS_PER_TASK, and calls gmx mdrun with -maxh set just below the SLURM wall clock so the job checkpoints before it is killed. You then resubmit the same script with -cpi to continue.

The laptop test run is the easy part. The first cluster submission is where most MSc students lose a week, because three things fail quietly at once: the module is not loaded inside the batch environment, the job asks for sixteen cores and then uses one, and the wall clock kills the run at hour twenty four with no usable restart point. None of those produce a helpful error. This guide covers the exact job script, how to size the request, how checkpointing interacts with the scheduler, and the real errors you will see. Written by the StemSkills Lab team, from 10+ years in sequence and structural bioinformatics, drug discovery and design, and multiscale molecular modeling.

What actually changes when you move GROMACS from a laptop to a cluster?

Four things change, and each one has a specific fix.

• Software is not on your PATH by default. Clusters use environment modules, so gmx exists only after a module load. Your interactive shell and the batch job do not share an environment.
• You must declare your resources in advance. SLURM gives you exactly what you asked for. Asking for one core and running sixteen threads is the most common reason a cluster run is slower than the laptop it came from.
• There is a hard wall clock. The job is terminated at the limit whether or not the simulation is finished, so the run has to be designed in restartable segments.
• The binary name may differ. Single-node builds give you gmx. MPI builds that span nodes usually give you gmx_mpi, launched under srun. Mixing them up is the second most common failure.

GROMACS is built for exactly this transition. The reference paper by Abraham and colleagues, GROMACS: High performance molecular simulations through multi-level parallelism from laptops to supercomputers (SoftwareX, 2015, vol. 1 to 2, pp. 19 to 25), describes the layered parallelism you are about to switch on. If you are still deciding which simulation package to learn first, start from our pillar guide to molecular dynamics with GROMACS.

How do you find and load the right GROMACS module?

Never assume a version. Ask the cluster what it has, in this order:

module avail gromacs
module spider gromacs        # on Lmod clusters, shows hidden versions and prerequisites
module purge
module load gromacs/2024.2   # substitute the exact name printed above
module list
gmx --version

module purge before module load matters more than it looks. It clears whatever your login profile happened to load and makes the job reproducible for anyone who reruns your script. gmx --version is the check that decides the rest of your script: it prints whether the build has MPI support, whether GPU support is compiled in, and the precision. If the header shows an MPI build, your binary is probably gmx_mpi and you launch it with srun.

Put every one of those module lines inside the job script. A module loaded in your interactive session does not carry into a batch job in any predictable way, and this single omission is what produces gmx: command not found in an otherwise perfect script.

What does a working GROMACS SLURM job script look like?

Here is a single node, single GPU production run. Every directive is documented in the official SLURM sbatch manual.

#!/bin/bash
#SBATCH --job-name=md_prod
#SBATCH --partition=gpu
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=1
#SBATCH --cpus-per-task=16
#SBATCH --gres=gpu:1
#SBATCH --time=24:00:00
#SBATCH --output=md_prod_%j.out
#SBATCH --error=md_prod_%j.err

module purge
module load gromacs/2024.2

export OMP_NUM_THREADS=${SLURM_CPUS_PER_TASK}

gmx mdrun -s prod.tpr -deffnm prod \
          -ntmpi 1 -ntomp ${SLURM_CPUS_PER_TASK} \
          -nb gpu -pme gpu -bonded gpu \
          -maxh 23.5 -cpi prod.cpt -noappend

Submit it with sbatch prod.sh, then watch it with squeue -u $USER and scontrol show job <jobid>.

For a run that spans more than one node you need the MPI build, and you must not pass -ntmpi, because thread-MPI and real MPI are alternatives rather than partners:

#!/bin/bash
#SBATCH --job-name=md_prod_mpi
#SBATCH --nodes=2
#SBATCH --ntasks-per-node=4
#SBATCH --cpus-per-task=8
#SBATCH --time=24:00:00
#SBATCH --output=md_prod_%j.out

module purge
module load gromacs/2024.2

export OMP_NUM_THREADS=${SLURM_CPUS_PER_TASK}

srun gmx_mpi mdrun -s prod.tpr -deffnm prod \
     -maxh 23.5 -cpi prod.cpt -noappend

That allocation is 2 nodes x 4 ranks x 8 threads, so 64 cores total, and the arithmetic has to match what you told SLURM. The GROMACS performance guide is explicit that “the number of ranks should be a multiple of the number of sockets, and the number of cores per node should be a multiple of the number of threads per rank”.

How many cores and threads should you request?

The honest answer is that more cores is not automatically faster, and the GROMACS manual gives a number for it: OpenMP multithreading scales well up to roughly 12 to 24 threads on Intel CPUs and 6 to 8 threads on AMD CPUs. Beyond that, adding threads per rank stops paying for itself and you are better off adding ranks.

The practical method is a short scaling test before you commit a week of allocation. Run the same 5,000 step .tpr at four different core counts, read the ns/day figure that mdrun prints at the end of the log, and pick the point where doubling the cores stops giving you close to double the throughput. That measurement takes under an hour and routinely saves days.

How do you stop the wall clock from killing your simulation?

You give mdrun its own deadline, slightly inside the scheduler’s. That is what -maxh is for. The GROMACS manual documents it as “Terminate after 0.99 times this time (hours)”, with a default of -1, meaning no limit at all unless you set one.

So -maxh 23.5 inside a --time=24:00:00 allocation tells mdrun to stop cleanly at about 23.3 hours, leaving roughly 40 minutes of slack. GROMACS writes a final checkpoint on that clean exit, which is the whole point. Without -maxh the job is killed mid-step and you fall back to whatever the last periodic checkpoint happened to be.

Those periodic checkpoints are controlled by -cpt, which defaults to 15 minutes. Fifteen minutes is a sensible default for most runs. Shorten it only if your filesystem is fast and your job is at real risk of a hard kill.

The manual is also honest about the limitation, describing -maxh as “useful to help cooperate with a job scheduler, but can be problematic if jobs can be suspended”. On a cluster that suspends and resumes jobs, wall-clock time keeps running while your simulation does not, so leave a wider margin there.

If your site sends a warning signal before the kill, --signal is worth adding. The SLURM manual describes it as: “When a job is within sig_time seconds of its end time, send it the signal sig_num.” It also warns that “the signal may be sent up to 60 seconds earlier than specified”, which is another reason not to run your margin down to nothing.

How do you restart a GROMACS run after the job hits the time limit?

You resubmit the identical script. The restart logic lives in one flag, and the GROMACS documentation describes its behaviour precisely: the -cpi option “is intelligent in the way that if no checkpoint file is found, GROMACS just assumes a normal run and starts from the first step of the .tpr file”.

That single sentence is why the same script works for both the first submission and every continuation. There is no separate restart script to maintain.

Two decisions matter here:

1. Append or not. By default mdrun appends to the existing outputs. With -noappend it instead writes “each output to a separate file, whose name includes a .partXXXX string”. Appending gives you tidy single files but fails awkwardly if a file was truncated. -noappend never fails, at the cost of stitching the pieces together afterwards. For unattended cluster runs, prefer -noappend.
2. Keep the checkpoint name consistent. If you set a custom -cpo name on the first run, you must pass that same name to -cpi on every restart.

When the run is finally done, join the parts:

gmx trjcat -f prod.part*.xtc -o prod_full.xtc
gmx eneconv -f prod.part*.edr -o prod_full.edr

How do you extend a simulation that finished but needs more time?

A completed 100 ns run that you now want to push to 200 ns is a different operation from a restart. You lengthen the run input, then continue from the checkpoint:

gmx convert-tpr -s prod.tpr -extend 100000 -o prod_extended.tpr
gmx mdrun -s prod_extended.tpr -cpi prod.cpt -deffnm prod_ext -maxh 23.5 -noappend

-extend takes picoseconds, so 100000 adds 100 ns. If you also need to change an mdp setting for the continuation, regenerate the tpr through grompp using the checkpoint as the state source with gmx grompp -f new.mdp -p topol.top -c original.gro -t state.cpt -o new.tpr, then run that.

Troubleshooting: the errors you will actually hit

• gmx: command not found in the SLURM output file. The module was loaded in your login shell, not in the job. Put module purge and module load inside the script, above the mdrun call.
• The cluster run is slower than your laptop. You almost certainly requested cores you never used. Check that OMP_NUM_THREADS is exported from SLURM_CPUS_PER_TASK, and read the mdrun log header, which prints the ranks and threads it actually started.
• mdrun refuses to append and reports that output files listed in the checkpoint are missing or too short. A previous job was killed mid-write. Rerun the continuation with -noappend and concatenate later.
• The job dies at the wall clock with no new checkpoint. -maxh was missing, or it was set equal to the SLURM limit instead of below it. The 0.99 factor is applied to your -maxh value, not to the scheduler’s limit, so it does not protect you on its own.
• The job never starts and sits pending. Compare your --time against the partition limit. SLURM documents that if the requested limit exceeds the partition’s limit, “the job will be left in a PENDING state (possibly indefinitely)”.
• A GPU flag is rejected or no GPU is detected. Confirm the partition actually has GPUs, that --gres=gpu:1 was granted, and that gmx --version reports GPU support compiled in. A CPU-only GROMACS build ignores nothing gracefully.
• mdrun warns that it could not set thread affinity. This is normal when you do not own the whole node. Leave -pin at its default of auto unless your allocation is a full exclusive node.

Frequently asked questions

Do I need a GPU to run GROMACS on a cluster?

No. GROMACS runs well on CPUs alone, and many university clusters have far more free CPU nodes than GPU nodes, so a CPU job often starts sooner. A GPU helps most for the non-bonded and PME work on larger solvated systems. Test both partitions with a short scaling run before committing.

Should I use gmx or gmx_mpi?

Use gmx with thread-MPI when everything fits on one node, which covers most student projects. Use gmx_mpi under srun only when you genuinely span nodes. The GROMACS manual notes that real MPI within a single node “runs more slowly than the thread-MPI version”.

How long should I set –time to?

Long enough to make progress, short enough to be scheduled. On a busy cluster a 12 hour job frequently starts hours before a 48 hour job. Because -cpi makes continuation free, a chain of shorter jobs usually finishes sooner in wall-clock terms than one very long request.

Is a checkpoint restart identical to an uninterrupted run?

A GROMACS checkpoint stores full-precision positions and velocities plus the state needed to restart the algorithms, so a continuation is a correct continuation of the same trajectory. Exact bitwise reproduction across different core counts or hardware is not guaranteed, which is expected for parallel MD and is not a defect in your run.

Can I run several short simulations in one job?

Yes, and it is often the efficient thing to do. Run replicates sequentially in one script, or use the GROMACS multi-simulation support to run them concurrently under one allocation. Either way, keep each replicate in its own directory so the default file names never collide.

If you are mapping out which of these skills to learn in what order, our computational biology skills roadmap sets the sequence, and the GROMACS pillar guide covers system preparation, equilibration and analysis around this step.