# Module environment (lmod)

<br>
<hr style="margin-right: 0px; margin-bottom: 4px; margin-left: 0px; margin-top: -24px; border:2px solid  #d9d9d9 "></hr>
<hr style="margin: 4px 0px; border:1px solid  #d9d9d9 "></hr>

## Short introduction 

---

HPC has a module step-system. To use some application, user need to follow these two steps, and insert applications into the search path:

1. In first step user determines the machine type (eg. amp, green or gray) by command:

       module load green/all        # for licensed programs   
    
    or
    
       module load green-spack      # for free programs  
       
2. User loads program needed by command:

       module load tau

The list of avalible modules can be looked by:

    module avail
       
where:  
<span style="color:red">
_**Lic** </span>  - a license is required, see user-guide for more information_  
<span style="color:red">
_**Uni** </span>  - commercial software with site licence, number of concurrent processes may be limited_  
<span style="color:red"> 
_**Reg** </span>  - registration required, see user-guide for more information_  
<span style="color:orange">
_**L**   </span> -  module is loaded_
<span style="color:blue">  
_**Dp**  </span> - deprecated (old modules, which have been superseeded by modules in the new structure)_
<span style="color:blue">  
_**O**   </span> - obsolete (module moved or superseeded by SPACK module)_  
<span style="color:blue">  
_**Exp**   </span> - Experimental module, used while testing software installation, module name may change or software may be deleted_  
_**D** - default Module._

<br>
<br>
<hr style="margin-right: 0px; margin-bottom: 4px; margin-left: 0px; margin-top: -24px; border:2px solid  #d9d9d9 "></hr>
<hr style="margin: 4px 0px; border:1px solid  #d9d9d9 "></hr>

## Long version 

---

The module system is used to manage settings for different applications. Many applications and libraries are not in the standard search path, this way it is possible to install two different versions of the same software/library that would otherwise create conflicts. The module system is used to insert applications into the search path (or remove them from it) on a per user and per occasion basis.

### Useful commands

 - All available modules can be looked through by command:

        module avail

    example output:

        ---------------------- /gpfs/mariana/modules/system ---------------------------
        amp-spack/0.17.1                                     (D)    green/all
        amp-spack/0.19.0                                            mpi/openmpi-x86_64           (D)
        amp/all                                                     mpi/openmpi3-x86_64
        experimental/all                                            oldmodules/all               (Dp)
        experimental/openmpi-4.1.1-gcc-10.3.0-newucx-noverbs        openmpi/4.1.1-gcc-4.8.5
        experimental/openmpi-4.1.1-gcc-10.3.0-newucx         (D)    openmpi/4.1.1-gcc-9.3-amp
        gray-spack/0.17.1                                           openmpi/4.1.1-gcc-9.3-amppmi
        gray/all                                                    openmpi/4.1.1-gcc-10.3.0     (D)
        green-spack/0.17.1                                   (D)    viz-spack/0.17.1
        green-spack/0.19.0                                          viz/all
        
        
        --------------------------- /etc/modulefiles ---------------------------
        mpi/openmpi-x86_64    mpi/openmpi3-x86_64


    modules are grouped in a hierarchy, there may be several versions of the same software installed, e.g. of the mpi library. Only one of these can be loaded at a single time. The default module of a group is marked by the `(D)`, if there is only one module in a group this is the default (unmarked).

 - To load a certain version of a module _(here - 0.19.0)_:

        module load green-spack/0.19.0

    To load the default module marked `(D)` _(here - green-spack/0.17.1)_:

        module load green-spack

 - Unloading a module _(here - green-spack)_:

        module unload green-spack/0.19.0

 -  Finding a module containing a certain part _(here - fem)_:

        module keyword fem

    Will be listed all modules that have _"FEM"_ in the description:


        The following modules match your search criteria: "fem"
        ---------------------------------------------------------------------------
    
        HPC2: HPC2/elmerfem-9.0
    
        elmerfem: elmerfem/9.0
    
        freefem: freefem/4.9-gcc-10.3.0-4ku6, freefem/4.9-gcc-10.3.0-6bjk
    
        mfem: mfem/4.3.0-gcc-10.3.0-lqmm, mfem/4.3.0-gcc-10.3.0-5i6k
    
        ---------------------------------------------------------------------------

 - To find out more about a specific module _(here - mfem)_:

        module spider mfem/4.3.0-gcc-10.3.0-5i6k

    gives

        You will need to load all module(s) on any one of the lines below before the 
        "mfem/4.3.0-gcc-10.3.0-5i6k" module is available to load.

        green-spack/0.17.1
 
        Help:
        Free, lightweight, scalable C++ library for finite element methods.

 - The `module whatis` command gives you a short explanation what the software is about, e.g.

        module whatis mercurial
        mercurial/5.8-gcc-10.3.0-python-3.8.12-mjum          : Mercurial is a free, distributed
        source control management tool. 

    or

        module whatis r
        r/4.1.1-gcc-10.3.0-zwgc       : R is 'GNU S', a freely available language and environment
        for statistical computing and graphics which provides a wide variety of statistical and 
        graphical techniques: linear and nonlinear modelling, statistical tests, time series 
        analysis, classification, clustering, etc. Please consult the R project homepage for 
        further information. 

Personal preferences and resources can be specified in the file `.modulerc.lua` and  `.bashrc` in the user's `$HOME` directory. For example it is possible to add a path for own module files for software installed by the user in the user's `$HOME` directory, 
automatically load some modules on login and to define one's own "default" modules using the entry "module_version("r/4.1.1-gcc-10.3.0-zwgc","default")" or introduce abbreviations using an entry like "module_alias("z13", "r/4.1.1-gcc-10.3.0-zwgc")" to define a module alias "z13".

examplele of `.modulerc.lua` file

    module_version("r/4.1.1-gcc-10.3.0-zwgc","default")
    module_alias("z13","r/4.1.1-gcc-10.3.0-zwgc")

    module_version("p/20.2-gcc-10.3.0-python-2.7.18-ij2m","default")
    module_alias("p20","p/20.2-gcc-10.3.0-python-2.7.18-ij2m")
    
examplele of `.bashrc` file
    
    # .bashrc

    # Source global definitions
    if [ -f /etc/bashrc ]; then
        . /etc/bashrc
    fi

    # Uncomment the following line if you don't like systemctl's auto-paging feature:
    # export SYSTEMD_PAGER=

    # User specific aliases and functions
    module load green-spack
    module load r/4.1.1-gcc-10.3.0-zwgc


<br>
<br> 
<hr style="margin-right: 0px; margin-bottom: 4px; margin-left: 0px; margin-top: -24px; border:2px solid  #d9d9d9 "></hr>
<hr style="margin: 4px 0px; border:1px solid  #d9d9d9 "></hr>

## Module groups

---

There are several module groups (directories) depending on the target architecture (green, gray, amp), operating system (CentOS, Ubuntu) and installation method (manual, spack). Software is optimized for the hardware when possible, that means "green" software (Skylake CPU) will not run on "gray" nodes (SandyBridge CPU) or "amp" (Zen2 CPU).

<span style="color:blue">
We moved to a new module structure! Modules from `/share/apps/modules` are being retired.

New modules are grouped, you can activate them by loading one or more of the following modules:

| modulegroup | description |
|-------------|-------------|
| green/all   | manually installed software (with possible optimization for green nodes) |
| gray/all   | manually installed software (with possible optimization for gray nodes, should run on green) |
| green-spack/0.17.1 | software installed with spack package manager (with possible optimization for green nodes) |
| gray-spack/0.17.1 | software installed with spack package manager (with possible optimization for gray nodes) |

<!---
Additionally, there will be science-field category directories with the most common software for that field to keep the module list short:

- /gpfs/mariana/modules/gray/engineering : for FEA, CFD
- /gpfs/mariana/modules/gray/chemistry : for chemistry software
- /gpfs/mariana/modules/gray/biology : for biology software (former bio directories)
- possibly more
--->
</span>

<br>
<br> 
<hr style="margin-right: 0px; margin-bottom: 4px; margin-left: 0px; margin-top: -24px; border:2px solid  #d9d9d9 "></hr>
<hr style="margin: 4px 0px; border:1px solid  #d9d9d9 "></hr>


## module use on **viz**

---

In order to make the module system work on **viz**, the following needs to be added to your `$HOME/.bashrc`


    if  [[ $(hostname -s) = viz ]]; then
       source /usr/share/lmod/6.6/init/bash
       module use /gpfs/mariana/modules/system
    fi



## Available modules

---

Currently the following modules are available, this serves as an example, please note that the list on this page will be updated very seldom, use `module avail` after login to get an up-to-date list of the availabe modules.


<details><summary>Click to expand</summary>

    --------------------------------------- /gpfs/mariana/modules/green/ALL ----------------------------------------
       MultiWFN/3.8              abaqus/2018          gcc/10.3.0              (D)    openmpi/4.1.1-gcc-10.3.0 (D)
       MultiWFN_noGUI/3.8        calculix/2.17        git/2.14.1                     orca/4.1.2
       R/4.0.2                   calculix/2.19 (D)    gnuplot/5.2.7                  orca/5.0.3               (D)
       R/4.1.1            (D)    elmerfem/9.0         octave/6.2.0                   salome/9.8.0
       Teem/1.11.0               gcc/5.2.0            openmpi/4.1.1-gcc-4.8.5        xtb-crest/6.4.1
    
    -------------------------------------- /gpfs/mariana/modules/green/spack ---------------------------------------
       bioawk/1.0-gcc-10.3.0-ge52
       chapel/1.24.1-gcc-10.3.0-kapp
       cp2k/8.2-gcc-10.3.0-7cv4
       cppzmq/4.7.1-gcc-10.3.0-y6ul
       cube/4.6-gcc-10.3.0-python-2.7.18-pc7m
       cubew/4.6-gcc-10.3.0-h3sm
       curl/7.79.0-gcc-10.3.0-sg6u
       dyninst/11.0.1-gcc-10.3.0-4fkp
       extrae/3.8.3-gcc-10.3.0-inni
       fastqc/0.11.9-gcc-10.3.0-o3jm
       freefem/4.9-gcc-10.3.0-6bjk
       gawk/5.1.0-gcc-10.3.0-b5dl
       gcc/10.3.0-gcc-10.3.0-qshu
       git/2.31.1-gcc-10.3.0-xl2q                        (D)
       gmsh/4.8.4-gcc-10.3.0-nija
       gnuplot/5.4.2-gcc-10.3.0-python-3.8.12-pcla       (D)
       go/1.17.2-gcc-10.3.0-5vfb
       gperftools/2.9.1-gcc-10.3.0-pjcu
       hdf5/1.10.7-gcc-10.3.0-c4ex
       hpctoolkit/2021.10.15-gcc-10.3.0-l5gd
       hpcviewer/2021.10-gcc-10.3.0-23zg
       jmol/14.31.0-gcc-10.3.0-2wul
       julia/1.6.3-gcc-10.3.0-tqfw
       llvm/13.0.0-gcc-10.3.0-dhdd
       lua-luajit/2.0.5-gcc-10.3.0-onhj
       lua/5.3.5-gcc-10.3.0-wt6f
       mawk/1.3.4-20171017-gcc-10.3.0-dvcb
       mercurial/5.8-gcc-10.3.0-python-3.8.12-mjum
       meshtool/oc8.1-gcc-10.3.0-k5ug
       mfem/4.3.0-gcc-10.3.0-5i6k
       molden/6.7-gcc-10.3.0-4ykf
       netcdf-c/4.8.1-gcc-10.3.0-qr5g
       netcdf-fortran/4.5.3-gcc-10.3.0-mvbu
       nwchem/7.0.2-gcc-10.3.0-openblas-cfuz
       octave/6.3.0-gcc-10.3.0-openblas-cskh
       octave/6.3.0-gcc-10.3.0-openblas-gjbt             (D)
       openblas/0.3.18-gcc-10.3.0-jmxf
       openfoam/1912-gcc-10.3.0-iic3
       openfoam/2112-gcc-10.3.0-37bt                     (D)
       openjdk/11.0.8_10-gcc-10.3.0-ve66
       openjdk/11.0.12_7-gcc-10.3.0-z6x7                 (D)
       otf/1.12.5salmon-gcc-10.3.0-wc2b
       otf2/2.3-gcc-10.3.0-6oki
       parallel-netcdf/1.12.2-gcc-10.3.0-ptkr
       perl/5.34.0-gcc-10.3.0-4rku
       py-jupyterlab/3.2.1-gcc-10.3.0-python-3.8.12-siuw
       py-pip/20.2-gcc-10.3.0-python-2.7.15-vlyo
       py-pip/20.2-gcc-10.3.0-python-2.7.18-ij2m
       py-pip/21.1.2-gcc-10.3.0-python-3.8.7-bj7d        (D)
       python/2.7.15-gcc-10.3.0-kjv6
       python/2.7.18-gcc-10.3.0-ckiy
       python/3.8.7-gcc-10.3.0-plhb                      (D)
       qcachegrind/20.12.2-gcc-10.3.0-python-3.8.12-shid
       r/4.1.1-gcc-10.3.0-zwgc
       ravel/1.0.0-gcc-10.3.0-python-3.8.12-pdvc
       rust/1.51.0-gcc-10.3.0-tj3j
       samtools/1.13-gcc-10.3.0-python-3.8.12-uwmu
       scalasca/2.6-gcc-10.3.0-a2p2
       scorep/7.0-gcc-10.3.0-dgia
       sortmerna/2017-07-13-gcc-10.3.0-p5a5
       star/2.7.6a-gcc-10.3.0-3u2w
       su2/7.2.0-gcc-10.3.0-python-3.8.12-sjl6
       tau/2.30.1-gcc-10.3.0-sx3x
       trimmomatic/0.39-gcc-10.3.0-qis6
       upcxx/2021.9.0-gcc-10.3.0-python-3.8.12-qt32
       valgrind/3.17.0-gcc-10.3.0-s2sc
       xeus/1.0.4-gcc-10.3.0-qr6f

</details>
<br/>


By default SPACK builds optimized for the CPU the software is build on.
The packages from the gray nodes will work on the green nodes but slower than the optimized modules. Conversely, the skylake-optimized modules will try to use hardware features not present on the gray nodes, so the software will not run there.

<!---
You can autoactivate the correct module path in your `.bashrc` with a code block like this:

    if [[ $(hostname -s) = base ]]; then
      module load green
      module load green-spack
    elif [[ $(hostname -s) = green* ]]; then
      module load green
      module load green-spack
    elif  [[ $(hostname -s) = gray* ]]; then
      module load gray
      module load gray-spack
    elif  [[ $(hostname -s) = mem1tb ]]; then
      module load gray
      module load gray-spack
    elif  [[ $(hostname -s) = amp* ]]; then
      module use /gpfs/mariana/modules/system
      module load amp
      module load amp-spack
    elif  [[ $(hostname -s) = viz ]]; then
      source /usr/share/lmod/6.6/init/bash
      module use /gpfs/mariana/modules/system
    fi
--->

<!--
### _SPACK for user build software_

SPACK can also be used by users to manage their own software stack inside their home directory (be aware, this takes a lot of space!).

see documentation on <https://spack.readthedocs.io/en/latest/>

SPACK includes GPU-offloading compiler for both Nvidia and AMD, profiling tools (Tau, HPCToolkit) and engineering simulation packages (ElmerFEM, OpenFOAM).

SPACK is used by University of Tartu, LRZ and HLRS.
-->

<!--
## Software modules from the EasyBuild package system

**under development, docs AND module positions/versions may change without notice**

EasyBuild is a package manager to install software packages. An advantage is to be able to relatively easily install consistent dependencies and multiple versions of a software. [List of packages](https://docs.easybuild.io/en/latest/version-specific/Supported_software.html)


EasyBuild is available on **amp** from AI lab

    module load amp


### _EasyBuild for user build software_

EasyBuild can also be used by users to manage their own software stack inside their home directory (be aware, this takes a lot of space!).

See documentation on [https://docs.easybuild.io/en/latest/](https://docs.easybuild.io/en/latest/)


EasyBuild seems to be more AI and Python oriented.


EasyBuild is used by AI-lab and will be used on LUMI. On LUMI it is "bring your own build recipe (.eb file)"!
-->

<br>
<br> 
<hr style="margin-right: 0px; margin-bottom: 4px; margin-left: 0px; margin-top: -24px; border:2px solid  #d9d9d9 "></hr>
<hr style="margin: 4px 0px; border:1px solid  #d9d9d9 "></hr>