Gaussian
Gaussian short introduction
Make gaussian.slurm batch script:
#!/bin/bash #SBATCH --job-name=Job_Name #SBATCH --mem-per-cpu=1GB #SBATCH --nodes=1 #SBATCH --ntasks=1 #SBATCH --cpus-per-task=24 #SBATCH -t 1-00:00:00 #SBATCH --partition=common #SBATCH --no-requeue module load green/all module load Gaussian/16.c01 SCRATCH=/state/partition1/$SLURM_JOB_ID export GAUSS_SCRDIR=$SCRATCH mkdir -p $SCRATCH g16 -m=24gb -p=24 < job.com > job.log #Clean after yourself rm -rf $SCRATCH
Copy job-input file job.com.
Submit the job on base:
sbatch gaussian.slurm
NB! More cores does not mean faster!!! See Benchmarks.
Check results using visualization software.
Gaussian long version
Gaussian is a general purpose package for calculation of electronic structures. It can calculate properties of molecules (structures, energies, spectroscopic and thermochemical properties, atomic charges, electron affinities and ionization potentials, electrostatic potentials and electron densities etc.) and reactions properties (such as reaction pathways, IRC) using different methods (such as Molecular mechanics, Semi-empirical methods, Hartree-Fock, Density functional, Møller-Plesset perturbation theory, coupled cluster). More about Gaussian features can be sound here.
Environment
There are currently two versions of Gaussian available on HPC: g09 (revision C.01) and g16 (revision C.01). Environment is set up by the commands:
module load green/all
module load Gaussian/16.c01
NB! For gray-ib partition gaussian16.c01.AVX should be loaded, for other partitions - gaussian16.c01.AVX2. More information on partition are in taltech user-guides.
Running Gaussian jobs
Gaussian input files are executed by the commands g09 or g16 depending on the version of Gaussian used. This command is usually placed in Slurm script.
g16 < job.com > job.log
Single core calculations
Gaussian by default execute jobs on only a single processor.
NB! If in a slurm script is dedined more processors, they will be reserved, but not utilizes.
Parallel jobs
Single-node (SMP) parallelism
To run multiple processors/cores job a number of cores should be specified. The number of cores can be defined via the -p flag (e.g. -p=8) in command line or by adding the %NprocShared command into Gaussian input file (e.g. %NprocShared=24). For more information see Gaussian manual. The number of processors requested should correspond to the number of processors requested in slurm script.
Example of slurm script for Gray partition:
#!/bin/bash
#SBATCH --job-name=Job_Name
#SBATCH --mem-per-cpu=1GB
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=4
#SBATCH -t 1-00:00:00
#SBATCH --partition=gray-ib
module load green/all
module load Gaussian/16.c01
SCRATCH=/state/partition1/$SLURM_JOB_ID
export GAUSS_SCRDIR=$SCRATCH
mkdir -p $SCRATCH
g16 < job.com > job.log
rm -rf $SCRATCH
Example of Gaussian input:
%Mem=4GB
%NprocShared=4
%chk=job.chk
#P B3LYP/6-311++G** Opt EmpiricalDispersion=GD3BJ
Job_Name
0,1
C 0.67650 0.42710 0.00022
H 0.75477 1.52537 0.00197
O 1.62208 -0.30498 -0.00037
S -1.01309 -0.16870 0.00021
H -1.58104 1.05112 -0.00371
Example of slurm script for Green partition:
#!/bin/bash
#SBATCH --job-name=Job_Name
##SBATCH --mem-per-cpu=1GB
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=4
#SBATCH -t 1-00:00:00
#SBATCH --partition=common
module load green/all
module load Gaussian/16.c01
SCRATCH=/state/partition1/$SLURM_JOB_ID
export GAUSS_SCRDIR=$SCRATCH
mkdir -p $SCRATCH
g16 -m=4gb -p=4 < job.com > job.log
rm -rf $SCRATCH
Example of Gaussian input:
%chk=job.chk
#P B3LYP/AUG-cc-pVTZ Opt Freq=(NoRaman)
HCOSH trans
0 1
C
H 1 1.1010569
O 1 1.1958524 2 123.6711058
O 1 1.4300000 3 122.8283301 2 -179.9280276
H 4 0.9600000 1 109.5000001 3 179.9228016
Memory
The default dynamic memory requested by Gaussian is frequently too small for successful job termination. Herein, if amount of memory requested is insufficient, the job can crash. There is no golden rule for memory requests. Usually, for common calculations (e.g. optimization, frequency etc.) 1-2 GB per 1 CPU is sufficient. This can be done by the -m flag in the command line (e.g. -m=48gb) or by adding the %Mem command in Gaussian input file (e.g. %Mem=1GB). For more information see Gaussian manual and taltech user-guides.
However, there are calculations that require more memory (e.g TD-DFT, large SCF calculations, etc.). Data from a slurm-JOBID.stat file can be useful to determine the amount of memory required for a computation. In slurm-JOBID.stat file the efficiency of memory utilization is shown.
Bad example:
Memory Utilized: 3.08 GB
Memory Efficiency: 11.83% of 26.00 GB
Good example:
Memory Utilized: 63.12 GB
Memory Efficiency: 98.62% of 64.00 GB
Time
Time limits depend on time partition used taltech user-guides. If the calculation time exceeds the time limit requested in the Slurm script, then the job will be killed. Therefore, it is recommended to request more time than is usually needed for calculation.
Using GPUs
GPUs are effective for larger molecules, DFT energies, gradients and frequencies calculations. GPUs are not effective for small jobs, MP2 or CCSD calculations.
GPU jobs can be run only on amp1 or amp2. To access amp user has to have ssh-keys copied to the base (how to do that).
Amp can be accessed by command:
ssh -X -Y -J uni-id@base.hpc.taltech.ee uni-id@amp
Each GPU must be controlled by a specific CPU, wherein, CPUs used as GPU controllers do not participate as compute nodes during the calculations.
The GPUs and CPUS used for calculations are specified with the %GPUCPU command, where gpu- and cpu-lists are a comma-separated lists, possibly including numerical ranges (e.g., 0-4,6). The corresponding items in the two lists are the GPU and its controlling CPU.
%cpu=0-9
%gpucpu=0-1=0-1
NB! Note that the controlling CPUs are included in %CPU command.
Example of gaussian-gpu.slurm script for amp:
#!/bin/bash
#SBATCH --job-name=Job_Name
#SBATCH -t 1:00:00
#SBATCH --no-requeue
#SBATCH --partition=gpu-test
#SBATCH --gres=gpu:A100:2
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=10
#SBATCH --mem=160GB
module use /gpfs/mariana/modules/system
module load green/all
module load Gaussian/16.c02
module use /illukas/software/modules
module load cuda/11.4
SCRATCH=/state/partition1/$SLURM_JOB_ID
export GAUSS_SCRDIR=$SCRATCH
mkdir -p $SCRATCH
g16 job.com > job.log
#Clean after yourself
rm -rf $SCRATCH
Example of Gaussian input job-gpu.com (bad example, since molecule is small):
%mem=160GB
%cpu=0-9
%gpucpu=0-1=0-1
# wb97xd/cc-pVDZ opt
GPU test
0 1
N 0.15134980 0.09540020 1.45819090
C 0.75130720 -1.21343470 1.83361500
H -0.39763650 0.48328420 2.23924210
H 0.89227330 0.78663430 1.26643050
C 2.08354180 -1.05427080 2.58684570
H 2.46028270 -2.07052410 2.83958100
H 1.89047670 -0.54642730 3.56027240
H 0.94017970 -1.74157910 0.87502690
C -0.27659390 -2.02386680 2.62322330
H -1.22744720 -2.09807790 2.06259160
H 0.09671630 -3.04868580 2.81503150
H -0.48448270 -1.55043170 3.60777900
C 3.18448030 -0.29819020 1.82025510
C 4.47118690 -0.16399430 2.65811660
C 3.50345720 -0.94054930 0.45675520
H 2.82725210 0.74302520 1.61940860
C 5.57163930 0.60375030 1.90697440
H 4.83763400 -1.18683360 2.90651280
H 4.24528280 0.32880900 3.62913610
C 4.59840130 -0.17643230 -0.29951390
H 3.82853470 -1.99290760 0.62773340
H 2.59267050 -0.99453480 -0.16920790
C 5.87589210 -0.03780040 0.54257470
H 6.49233690 0.65984540 2.52562860
H 5.23630520 1.65412070 1.74873850
H 4.81114970 -0.67947290 -1.26682010
H 4.21177720 0.83532620 -0.55856640
H 6.64325110 0.55322730 -0.00130230
H 6.31606730 -1.04822960 0.70571860
Allocation of memory
Allocating sufficient amounts of memory for GPU jobs is even more important when for CPU jobs. GPUs can have up to 16 GB of memory, wherein, must be at least an equal amount of memory given to the GPU and each control CPU thread.
Gaussian gives equal shares of memory to each thread, this means that the total memory allocated should be the number of threads times the memory required. Allocating Gaussian 11-12 GB is appropriate for a 16 GB GPU (see example above).
Restarting a failed/interrupted calculation
Killed or failed jobs can be restarted, but for this checkpoint file should be generated via a %Chk command within the Gaussian input file. For more information see Gaussian FAQ, Gaussian restart and Using Gaussian Checkpoint Files.
NB! Checkpoint files are very heavy and are readable only on the machine on which they were generated. After successful completion of the calculation, it is recommended to delete these files.
How to cite:
Gaussian 16 - https://gaussian.com/citation/
Gaussian 09 - Gaussian 09, Revision C.01, Frisch, M.J.; Trucks, G.W.; Schlegel, H.B.; Scuseria, G.E.; Robb, M.A.; Cheeseman, J.R.; Scalmani, G.; Barone, V.; Mennucci, B.; Petersson, G.A.; Nakatsuji, H.; Caricato, M.; Li, X.; Hratchian, H.P.; Izmaylov, A.F.; Bloino, J.; Zheng, G.; Sonnenberg, J.L.; Hada, M.; Ehara, M.; Toyota, K.; Fukuda, R.; Hasegawa, J.; Ishida, M.; Nakajima, T.; Honda, Y.; Kitao, O.; Nakai, H.; Vreven, T.; Montgomery, J.A., Jr.; Peralta, J.E.; Ogliaro, F.; Bearpark, M.; Heyd, J.J.; Brothers, E.; Kudin, K.N.; Staroverov, V.N.; Kobayashi, R.; Normand, J.; Raghavachari, K.; Rendell, A.; Burant, J.C.; Iyengar, S.S.; Tomasi, J.; Cossi, M.; Rega, N.; Millam, N.J.; Klene, M.; Knox, J.E.; Cross, J.B.; Bakken, V.; Adamo, C.; Jaramillo, J.; Gomperts, R.; Stratmann, R.E.; Yazyev, O.; Austin, A. J.; Cammi, R.; Pomelli, C.; Ochterski, J. W.; Martin, R.L.; Morokuma, K.; Zakrzewski, V.G.; Voth, G.A.; Salvador, P.; Dannenberg, J.J.; Dapprich, S.; Daniels, A.D.; Farkas, Ö.; Foresman, J.B.; Ortiz, J.V.; Cioslowski, J.; Fox, D.J. Gaussian, Inc., Wallingford CT, 2009.
Benchmarks for parallel jobs
Gaussian example benchmarks performed with Gaussian 16 C01 AVX2.
The job had 82 atoms.
