Compiling the ``kaiju`` software on CentOS-Stream 9
=======================================================

Introduction
------------

These instructions will walk you through the process of building and installing the ``kaiju`` software on a CentOS-Stream 9 system.

These instructions assume that the user is using the ``bash`` shell, and that no modifications have been made to the user "dotfiles" (``$HOME/.bashrc``, ``$HOME/.bash_profile``). If you have customized either of these files for your account, please carefully save and inspect the output from each command in the build process to ensure that no unexpected problems have crept in. To facilitate this practice, all of the commands shown below will illustrate how to save command output, and how to measure how long each step takes. The latter is a useful bit of information which can help identify build problems early in the process, avoiding much wasted time and effort later.

In the instructions below, all code will be built and installed in a CentOS-Stream 9-specific subdirectory of the user home directory, i.e. ``$HOME/centos-stream-9``. This particular organization is not required - it is intended as an example of one possible way to segregate software that has been built for multiple systems.

Building the serial version of the ``kaiju`` software on CentOS-Stream 9
----------------------------------------------------------------------------

Step 1: Configure build tools
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Begin by configuring tools and libraries needed for the build:

.. code-block:: shell

   # Add the Intel compilers to PATH.
   source /opt/intel/oneapi/setvars.sh

   # Set variables for HDF5.
   export HOST_SYSTEM=centos-stream-9
   export HDF5_DIR=$HOME/$HOST_SYSTEM/local/hdf5/1.14.1-2
   export PATH=$HDF5_DIR/bin:$PATH
   export HDF5_INCLUDE_DIRS=$HDF5_DIR/include
   export HDF5_LIBRARIES=$HDF5_DIR/lib
   export CPATH=$HDF5_INCLUDE_DIRS
   export INCLUDE="-I$HDF5_INCLUDE_DIRS"

   # Configure CDF.
   source $HOME/$HOST_SYSTEM/local/cdf/3.9.0/bin/definitions.B


Step 2: Create the build directory
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Create a system-specific build directory.

.. code-block:: shell

   cd $HOME
   mkdir HOST_SYSTEM
   cd $HOST_SYSTEM


Then make an additional subdirectory level for the branch of the code you are building (the ``development`` branch is used as an example). This arrangement is useful when you need to maintain simultaneous builds of different branches.

code-block:: shell

   export KAIJU_BRANCH_NAME=development
   mkdir $KAIJU_BRANCH_NAME
   cd $KAIJU_BRANCH_NAME


Step 3: Clone the ``kaiju`` repository from BitBucket
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

NOTE: This step assumes you have been granted access to the ``kaiju`` repository on BitBucket, and that you have configured an SSH key pair for use with BitBucket. If you need help with these tasks, please contact a CGS team member for assistance.

Clone the ``kaiju`` repository (or "repo") from BitBucket:

.. code-block:: shell

   git clone git@bitbucket.org:aplkaiju/kaiju.git


This process should take a minute or so. When complete, verify that the ``kaiju`` code exists in your directory (the actual directory contents may differ slightly from what is shown below):

.. code-block:: shell

   ls kaiju
   analysis  cmake  CMakeLists.txt  examples  external  gitHookScripts  kaiju.sublime-project  kaipy  places  pytests  quickstart  README.md  scripts  setup.py  src  testingScripts  tests  xml


Now move down into the cloned repo, and switch to the branch of the code you wish to use. By default, the cloned repository provides the ``master`` branch, but we want the ``development`` branch:

.. code-block:: shell

   cd kaiju
   git switch $KAIJU_BRANCH_NAME


Step 4: Run ``cmake`` to create the ``Makefile`` needed to build the software
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Since the ``kaiju`` code can be built in serial and MPI forms, we first make a directory in which to build the serial version of the code (use whatever name you prefer, but ``build_serial`` is simple and unambiguous):

.. code-block:: shell

   export KAIJU_BUILD_NAME=build_serial
   KAIJU_BUILD_PATH=$KAIJU_HOME/$KAIJU_BUILD_NAME
   mkdir -p $KAIJU_BUILD_PATH
   cd $KAIJU_BUILD_PATH


Now run the ``cmake`` command. Save the ``cmake`` output, and use timestamps for each step. The options shown below direct the build process to use a recent version of the Intel Fortran compiler:

.. code-block:: shell

   date; time FC=`which ifort` cmake -DALLOW_INVALID_COMPILERS=ON >& cmake.out; date


This command usually takes 2-3 seconds, depending on system activity. Examine the output file ``cmake.out`` for problems. It *should* look something like this:

.. code-block::

   -- The Fortran compiler identification is Intel 2021.10.0.20230609
   -- Detecting Fortran compiler ABI info
   -- Detecting Fortran compiler ABI info - done
   -- Check for working Fortran compiler: /opt/intel/oneapi/compiler/2023.2.0/linux/bin/intel64/ifort - skipped
   -- Checking whether /opt/intel/oneapi/compiler/2023.2.0/linux/bin/intel64/ifort supports Fortran 90
   -- Checking whether /opt/intel/oneapi/compiler/2023.2.0/linux/bin/intel64/ifort supports Fortran 90 - yes
   -- Found HDF5: /home/ewinter/centos-stream-9/local/hdf5/1.14.1-2/lib/libhdf5_fortran.so;/home/ewinter/centos-stream-9/local/hdf5/1.14.1-2/lib/libhdf5.so;/usr/lib64/libz.so;/usr/lib64/libdl.a;/usr/lib64/libm.so (found version "1.14.1-2") found components: Fortran 
   -- Found OpenMP_Fortran: -qopenmp (found version "5.0") 
   -- Found OpenMP: TRUE (found version "5.0") found components: Fortran 
   CMake Warning at cmake/compilers.cmake:61 (message):
     Setting default optimization to O2 to avoid certain Intel compiler bugs
   Call Stack (most recent call first):
     CMakeLists.txt:99 (include)


   -------------------------
   Configuration summary ...
   System: localhost.localdomain
   OS: Linux
   Processor: x86_64
   Compiler: Intel / 2021.10.0.20230609
             /opt/intel/oneapi/compiler/2023.2.0/linux/bin/intel64/ifort
   HDF5 Wrapper: /home/ewinter/centos-stream-9/local/hdf5/1.14.1-2/bin/h5fc
   Version: 3f4e147c / development
   Build Type: Release
   Base Flags:  -fPIC -free -implicitnone -qopenmp
   Build Flags: -O2 -align array64byte -align rec32byte -no-prec-div -fast-transcendentals -ipo
   -------------------------

   Adding CHIMP module ...
       EB IC file is /home/ewinter/centos-stream-9/cgs/kaiju/development/kaiju/src/chimp/ebICs/ebICstd.F90
       TP IC file is /home/ewinter/centos-stream-9/cgs/kaiju/development/kaiju/src/chimp/tpICs/tpICstd.F90
       Adding executable project.x
       Adding executable psd.x
       Adding executable push.x
       Adding executable slice.x
       Adding executable chop.x
       Adding executable trace.x
       Adding executable sctrack.x
       Adding executable calcdb.x
       Adding executable wpicheck.x
   Adding Gamera module ...
       Bricksize is 16
       IC file is /home/ewinter/centos-stream-9/cgs/kaiju/development/kaiju/src/gamera/ICs/null.F90
       Adding executable gamera.x
   Adding Gamera Helio module ...
       IC file is /home/ewinter/centos-stream-9/cgs/kaiju/development/kaiju/src/gamera/ICs/helio/wsa.F90
       Adding executable gamhelio.x
   Adding ReMIX module ...
       Adding executable remix.x
   Adding RCM module ...
       RCM Grid is of size 180 x 361 x 160
       Adding executable rcm.x
   Adding Voltron module ...
       IC file is /home/ewinter/centos-stream-9/cgs/kaiju/development/kaiju/src/voltron/ICs/earthcmi.F90
       Adding executable voltron.x
   -- Configuring done
   -- Generating done
   -- Build files have been written to: /home/ewinter/centos-stream-9/cgs/kaiju/development/kaiju/build_serial

Step 5: Compile the ``kaiju`` software
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Now use ``make`` to build the ``kaiju`` software, time-stamping and saving the output:

.. code-block:: shell

   date; time make >& make.out; date


This command should complete in about 8-9 minutes on a CentOS-Stream 9 system running on an i9 processor. When the command is finished, check the output file ``make.out``. The file is long, but the last few lines should look something like this:

.. code-block::

   [ 99%] Linking Fortran executable bin/sctrack.x
   [ 99%] Built target sctrack.x
   [ 99%] Built target chimp
   [ 99%] Built target voltron
   Scanning dependencies of target remix.x
   [100%] Building Fortran object CMakeFiles/remix.x.dir/src/drivers/remix.F90.o
   [100%] Linking Fortran executable bin/remix.x
   [100%] Built target remix.x
   [100%] Built target remix
   [100%] Built target gamera

To verify that all of the ``kaiju`` programs have been built, examine the ``bin`` subdirectory of your ``build_serial`` directory (this list will evolve as more programs are added):

.. code-block:: shell

   ls bin
   calcdb.x  gamhelio.x  psd.x   remix2rcm.x    sctrack.x  voltron.x
   chop.x    kaitoy.x    push.x  remix2remix.x  slice.x    wpicheck.x
   gamera.x  project.x   rcm.x   remix.x        trace.x


Using the ``kaiju`` software
--------------------------------

Once built, you must run the setup script before using the ``kaiju`` software:

.. code-block:: shell

   source $KAIJU_HOME/scripts/setupEnvironment.sh


This script will set environment variables needed by the ``kaiju`` software, including the ``KAIJUHOME`` environment variable (not the ``KAIJU_HOME`` environment variable). However, the path to the compiled programs is not added - you will need to specify the complete path when using compiled programs. For example,  to run the serial version of ``gamera.x``:

.. code-block:: shell

   $KAIJUHOME/build_serial/bin/gamera.x
