Module environment (lmod)




Short introduction


HPC has a module step-system. To use some application, user need to follow these two steps, and insert applications into the search path:

  1. In first step user determines the machine type (eg. amp, green or gray) by command:

    module load green/all        # for licensed programs   
    

    or

    module load green-spack      # for free programs  
    
  2. User loads program needed by command:

    module load tau
    

The list of avalible modules can be looked by:

module avail

where:
Lic - a license is required, see user-guide for more information
Uni - commercial software with site licence, number of concurrent processes may be limited
Reg - registration required, see user-guide for more information
L - module is loaded
Dp
- deprecated (old modules, which have been superseeded by modules in the new structure)
O
- obsolete (module moved or superseeded by SPACK module)

Exp
- Experimental module, used while testing software installation, module name may change or software may be deleted
D - default Module.





Long version


The module system is used to manage settings for different applications. Many applications and libraries are not in the standard search path, this way it is possible to install two different versions of the same software/library that would otherwise create conflicts. The module system is used to insert applications into the search path (or remove them from it) on a per user and per occasion basis.

Useful commands

  • All available modules can be looked through by command:

     module avail
    

    example output:

     ---------------------- /gpfs/mariana/modules/system ---------------------------
     amp-spack/0.17.1                                     (D)    green/all
     amp-spack/0.19.0                                            mpi/openmpi-x86_64           (D)
     amp/all                                                     mpi/openmpi3-x86_64
     experimental/all                                            oldmodules/all               (Dp)
     experimental/openmpi-4.1.1-gcc-10.3.0-newucx-noverbs        openmpi/4.1.1-gcc-4.8.5
     experimental/openmpi-4.1.1-gcc-10.3.0-newucx         (D)    openmpi/4.1.1-gcc-9.3-amp
     gray-spack/0.17.1                                           openmpi/4.1.1-gcc-9.3-amppmi
     gray/all                                                    openmpi/4.1.1-gcc-10.3.0     (D)
     green-spack/0.17.1                                   (D)    viz-spack/0.17.1
     green-spack/0.19.0                                          viz/all
    
    
     --------------------------- /etc/modulefiles ---------------------------
     mpi/openmpi-x86_64    mpi/openmpi3-x86_64
    

    modules are grouped in a hierarchy, there may be several versions of the same software installed, e.g. of the mpi library. Only one of these can be loaded at a single time. The default module of a group is marked by the (D), if there is only one module in a group this is the default (unmarked).

  • To load a certain version of a module (here - 0.19.0):

     module load green-spack/0.19.0
    

    To load the default module marked (D) (here - green-spack/0.17.1):

     module load green-spack
    
  • Unloading a module (here - green-spack):

     module unload green-spack/0.19.0
    
  • Finding a module containing a certain part (here - fem):

    module keyword fem
    

    Will be listed all modules that have “FEM” in the description:

    The following modules match your search criteria: "fem"
    ---------------------------------------------------------------------------
    
    HPC2: HPC2/elmerfem-9.0
    
    elmerfem: elmerfem/9.0
    
    freefem: freefem/4.9-gcc-10.3.0-4ku6, freefem/4.9-gcc-10.3.0-6bjk
    
    mfem: mfem/4.3.0-gcc-10.3.0-lqmm, mfem/4.3.0-gcc-10.3.0-5i6k
    
    ---------------------------------------------------------------------------
    
  • To find out more about a specific module (here - mfem):

     module spider mfem/4.3.0-gcc-10.3.0-5i6k
    

    gives

     You will need to load all module(s) on any one of the lines below before the 
     "mfem/4.3.0-gcc-10.3.0-5i6k" module is available to load.
    
     green-spack/0.17.1
    
     Help:
     Free, lightweight, scalable C++ library for finite element methods.
    
  • The module whatis command gives you a short explanation what the software is about, e.g.

     module whatis mercurial
     mercurial/5.8-gcc-10.3.0-python-3.8.12-mjum          : Mercurial is a free, distributed
     source control management tool. 
    

    or

     module whatis r
     r/4.1.1-gcc-10.3.0-zwgc       : R is 'GNU S', a freely available language and environment
     for statistical computing and graphics which provides a wide variety of statistical and 
     graphical techniques: linear and nonlinear modelling, statistical tests, time series 
     analysis, classification, clustering, etc. Please consult the R project homepage for 
     further information. 
    

Personal preferences and resources can be specified in the file .modulerc.lua and .bashrc in the user’s $HOME directory. For example it is possible to add a path for own module files for software installed by the user in the user’s $HOME directory, automatically load some modules on login and to define one’s own “default” modules using the entry “module_version(”r/4.1.1-gcc-10.3.0-zwgc”,”default”)” or introduce abbreviations using an entry like “module_alias(”z13”, “r/4.1.1-gcc-10.3.0-zwgc”)” to define a module alias “z13”.

examplele of .modulerc.lua file

module_version("r/4.1.1-gcc-10.3.0-zwgc","default")
module_alias("z13","r/4.1.1-gcc-10.3.0-zwgc")

module_version("p/20.2-gcc-10.3.0-python-2.7.18-ij2m","default")
module_alias("p20","p/20.2-gcc-10.3.0-python-2.7.18-ij2m")

examplele of .bashrc file

# .bashrc

# Source global definitions
if [ -f /etc/bashrc ]; then
    . /etc/bashrc
fi

# Uncomment the following line if you don't like systemctl's auto-paging feature:
# export SYSTEMD_PAGER=

# User specific aliases and functions
module load green-spack
module load r/4.1.1-gcc-10.3.0-zwgc




Module groups


There are several module groups (directories) depending on the target architecture (green, gray, amp), operating system (CentOS, Ubuntu) and installation method (manual, spack). Software is optimized for the hardware when possible, that means “green” software (Skylake CPU) will not run on “gray” nodes (SandyBridge CPU) or “amp” (Zen2 CPU).

We moved to a new module structure! Modules from `/share/apps/modules` are being retired.

New modules are grouped, you can activate them by loading one or more of the following modules:

modulegroup description
green/all manually installed software (with possible optimization for green nodes)
gray/all manually installed software (with possible optimization for gray nodes, should run on green)
green-spack/0.17.1 software installed with spack package manager (with possible optimization for green nodes)
gray-spack/0.17.1 software installed with spack package manager (with possible optimization for gray nodes)




module use on viz


In order to make the module system work on viz, the following needs to be added to your $HOME/.bashrc

if  [[ $(hostname -s) = viz ]]; then
   source /usr/share/lmod/6.6/init/bash
   module use /gpfs/mariana/modules/system
fi

Available modules


Currently the following modules are available, this serves as an example, please note that the list on this page will be updated very seldom, use module avail after login to get an up-to-date list of the availabe modules.

Click to expand
--------------------------------------- /gpfs/mariana/modules/green/ALL ----------------------------------------
   MultiWFN/3.8              abaqus/2018          gcc/10.3.0              (D)    openmpi/4.1.1-gcc-10.3.0 (D)
   MultiWFN_noGUI/3.8        calculix/2.17        git/2.14.1                     orca/4.1.2
   R/4.0.2                   calculix/2.19 (D)    gnuplot/5.2.7                  orca/5.0.3               (D)
   R/4.1.1            (D)    elmerfem/9.0         octave/6.2.0                   salome/9.8.0
   Teem/1.11.0               gcc/5.2.0            openmpi/4.1.1-gcc-4.8.5        xtb-crest/6.4.1

-------------------------------------- /gpfs/mariana/modules/green/spack ---------------------------------------
   bioawk/1.0-gcc-10.3.0-ge52
   chapel/1.24.1-gcc-10.3.0-kapp
   cp2k/8.2-gcc-10.3.0-7cv4
   cppzmq/4.7.1-gcc-10.3.0-y6ul
   cube/4.6-gcc-10.3.0-python-2.7.18-pc7m
   cubew/4.6-gcc-10.3.0-h3sm
   curl/7.79.0-gcc-10.3.0-sg6u
   dyninst/11.0.1-gcc-10.3.0-4fkp
   extrae/3.8.3-gcc-10.3.0-inni
   fastqc/0.11.9-gcc-10.3.0-o3jm
   freefem/4.9-gcc-10.3.0-6bjk
   gawk/5.1.0-gcc-10.3.0-b5dl
   gcc/10.3.0-gcc-10.3.0-qshu
   git/2.31.1-gcc-10.3.0-xl2q                        (D)
   gmsh/4.8.4-gcc-10.3.0-nija
   gnuplot/5.4.2-gcc-10.3.0-python-3.8.12-pcla       (D)
   go/1.17.2-gcc-10.3.0-5vfb
   gperftools/2.9.1-gcc-10.3.0-pjcu
   hdf5/1.10.7-gcc-10.3.0-c4ex
   hpctoolkit/2021.10.15-gcc-10.3.0-l5gd
   hpcviewer/2021.10-gcc-10.3.0-23zg
   jmol/14.31.0-gcc-10.3.0-2wul
   julia/1.6.3-gcc-10.3.0-tqfw
   llvm/13.0.0-gcc-10.3.0-dhdd
   lua-luajit/2.0.5-gcc-10.3.0-onhj
   lua/5.3.5-gcc-10.3.0-wt6f
   mawk/1.3.4-20171017-gcc-10.3.0-dvcb
   mercurial/5.8-gcc-10.3.0-python-3.8.12-mjum
   meshtool/oc8.1-gcc-10.3.0-k5ug
   mfem/4.3.0-gcc-10.3.0-5i6k
   molden/6.7-gcc-10.3.0-4ykf
   netcdf-c/4.8.1-gcc-10.3.0-qr5g
   netcdf-fortran/4.5.3-gcc-10.3.0-mvbu
   nwchem/7.0.2-gcc-10.3.0-openblas-cfuz
   octave/6.3.0-gcc-10.3.0-openblas-cskh
   octave/6.3.0-gcc-10.3.0-openblas-gjbt             (D)
   openblas/0.3.18-gcc-10.3.0-jmxf
   openfoam/1912-gcc-10.3.0-iic3
   openfoam/2112-gcc-10.3.0-37bt                     (D)
   openjdk/11.0.8_10-gcc-10.3.0-ve66
   openjdk/11.0.12_7-gcc-10.3.0-z6x7                 (D)
   otf/1.12.5salmon-gcc-10.3.0-wc2b
   otf2/2.3-gcc-10.3.0-6oki
   parallel-netcdf/1.12.2-gcc-10.3.0-ptkr
   perl/5.34.0-gcc-10.3.0-4rku
   py-jupyterlab/3.2.1-gcc-10.3.0-python-3.8.12-siuw
   py-pip/20.2-gcc-10.3.0-python-2.7.15-vlyo
   py-pip/20.2-gcc-10.3.0-python-2.7.18-ij2m
   py-pip/21.1.2-gcc-10.3.0-python-3.8.7-bj7d        (D)
   python/2.7.15-gcc-10.3.0-kjv6
   python/2.7.18-gcc-10.3.0-ckiy
   python/3.8.7-gcc-10.3.0-plhb                      (D)
   qcachegrind/20.12.2-gcc-10.3.0-python-3.8.12-shid
   r/4.1.1-gcc-10.3.0-zwgc
   ravel/1.0.0-gcc-10.3.0-python-3.8.12-pdvc
   rust/1.51.0-gcc-10.3.0-tj3j
   samtools/1.13-gcc-10.3.0-python-3.8.12-uwmu
   scalasca/2.6-gcc-10.3.0-a2p2
   scorep/7.0-gcc-10.3.0-dgia
   sortmerna/2017-07-13-gcc-10.3.0-p5a5
   star/2.7.6a-gcc-10.3.0-3u2w
   su2/7.2.0-gcc-10.3.0-python-3.8.12-sjl6
   tau/2.30.1-gcc-10.3.0-sx3x
   trimmomatic/0.39-gcc-10.3.0-qis6
   upcxx/2021.9.0-gcc-10.3.0-python-3.8.12-qt32
   valgrind/3.17.0-gcc-10.3.0-s2sc
   xeus/1.0.4-gcc-10.3.0-qr6f

By default SPACK builds optimized for the CPU the software is build on. The packages from the gray nodes will work on the green nodes but slower than the optimized modules. Conversely, the skylake-optimized modules will try to use hardware features not present on the gray nodes, so the software will not run there.