AtHeartEngineer/kaiju
1
0
Fork 0
mirror of https://github.com/JHUAPL/kaiju.git synced 2026-01-08 22:58:05 -05:00
Code Issues Packages Projects Releases Wiki Activity

GTR updates

Browse Source
This commit is contained in:
cookieenick
2025-06-14 22:10:05 -06:00
parent 726e091da8
commit 570ac7546c
18 changed files with 1624 additions and 51 deletions
Download Patch File Download Diff File Expand all files Collapse all files

1
.gitignore vendored
View File

@@ -50,3 +50,4 @@ Thumbs.db
# Built documentation from Sphinx.
docs/build/
docs/_build/

29
docs/source/building/buildDerecho.rst
View File

@@ -37,13 +37,33 @@ module set:
.. code-block:: bash
module --force purge
module load ncarenv/23.06
module load ncarenv/23.09
module load cmake/3.26.3
module load craype/2.7.20
module load intel/2023.0.0
module load ncarcompilers/1.0.0
module load cray-mpich/8.1.25
module load hdf5-mpi/1.12.2
module load conda/latest
.. note::
For Geospace runs coupled with TIE-GCM (GTR), use the following modules:
.. code-block:: bash
module --force purge
module load ncarenv/23.09
module load cmake/3.26.3
module load craype/2.7.31
module load intel-classic/2023.2.1
module load cray-mpich/8.1.27
module load ncarcompilers/1.0.0
module load mkl/2023.2.0
module load hdf5-mpi/1.12.2
module load netcdf-mpi/4.9.2
module load esmf/8.6.0
module load conda/latest
.. important::
@@ -53,7 +73,6 @@ module set:
**11 April 2025**, and is subject to change as the compute environment
changes.
Build the ``kaiju`` software
----------------------------
@@ -78,13 +97,13 @@ build directory in any convenient location.
# You can pick one compile target below or compile all of them, if you'd like
# Compile the MAGE model for geospace simulations
make voltron_mpi.x >& make-voltron.out
make -j4 voltron_mpi.x >& make-voltron.out
# Compile the GAMERA-helio model for inner heliosphere simulations
make gamhelio_mpi.x >& make-gamhelio.out
make -j4 gamhelio_mpi.x >& make-gamhelio.out
# Compile analysis tools
make calcdb.x chop.x sctrack.x slice.x >& make-analysis.out
make -j4 calcdb.x chop.x sctrack.x slice.x >& make-analysis.out
When finished, your build directory will contain a ``bin``

42
docs/source/building/buildPleiades.rst
View File

@@ -26,11 +26,53 @@ module set:
.. code-block:: bash
module --force purge
module use -a /swbuild/analytix/tools/modulefile
module load nas
module load pkgsrc/2022Q1-rome
module load comp-intel/2020.4.304
module load mpi-hpe/mpt.2.23
module load hdf5/1.8.18_mpt
module load miniconda3/v4
.. note::
For Geospace runs coupled with TIE-GCM (GTR), use the following modules:
.. warning::
The GTR currently required custom built NetCDF and ESMF modules on ``pleiades``. If you need to
run GTR, you will need access to ``/home7/nrao3/local3`` and ``/nobackup/nrao3/tiegcm/tiegcm3.0/data``,
please reach out to ``nikhilr@ucar.edu`` with the following:
- Your Pleiades username
- Your Name
- Your Institution
.. code-block:: bash
module --force purge
module use -a /nasa/modulefiles/testing
module use -a /swbuild/analytix/tools/modulefile
module load nas
module load comp-intel/2020.4.304
module load mpi-hpe/mpt.2.30
module load szip/2.1.1
module load hdf5/1.12.3_mpt
module load miniconda3/v4
export FC=mpif90
export CC=mpicc
export CXX=mpicxx
export PREFIX=/home7/nrao3/local3
export LIBRARY_PATH=${LIBRARY_PATH}:$PREFIX/lib
export LD_LIBRARY_PATH=$LIBRARY_PATH
export CPATH=$PREFIX/include
export PATH=${PATH}:$PREFIX/bin
.. important::

497
docs/source/makeitso/engage.rst Normal file
View File

@@ -0,0 +1,497 @@
Engage
===============
Introduction
------------
The Python script ``engage.py`` was developed to simplify the process of
configuring and running GTR MAGE (that is, the geospace application of the ``kaiju`` software.) It
provides an interactive, prompt-driven interface to specify all of the
parameters needed for a model run.
The ``engage.py`` script is a wrapper around the ``makeitso.py`` and TIE-GCM's ``tiegcmrun``
script, which is used to prepare the necessary files for a GTR MAGE model run.
- For more details on the ``makeitso.py`` script, see the :doc:`makeitso </makeitso/makeitso>` documentation.
- For more details on the TIE-GCM ``tiegcmrun`` script, see the `tiegcmrun <https://tiegcm-docs.readthedocs.io/en/latest/tiegcm/quickstart>`_ documentation.
The ``engage.py`` script can operate in one of three different modes:
``BASIC``, ``INTERMEDIATE``, or ``EXPERT``. Each mode provides access to
a subset of the ``kaiju`` and ``tiegcm`` parameters.
* The ``BASIC`` mode
Requires the user to provide the minimum set of parameters needed to specify a model
run, such as the run ID, and the simulation time periods.
* The ``INTERMEDIATE`` mode
Allows the user to specify all of the
parameters from the ``BASIC`` mode, as well as a wider set of run parameters,
such as non-standard file locations and some MHD parameters and TIE-GCM parameters.
* The ``EXPERT`` mode
Provides access to all of the user-adjustable
parameters from the ``kaiju`` and ``TIE-GCM`` software.
When finished, the script generates the files needed to run a magnetosphere model, and saves
all options in a convenient JSON file so that the run can be repeated at a
later date.
Running the ``engage.py`` script
----------------------------------
The ``engage.py`` script is provided as part of the ``kaiju`` software. It
is found at ``$KAIJUHOME/scripts/makeitso/engage.py``, where ``$KAIJUHOME``
is the location of your ``kaiju`` software tree. After configuring your
``kaiju`` software, you can get help text for the script like this:
.. code-block:: bash
engage.py --help
usage: engage.py [-h] [--clobber] [--debug] [--mode MODE] [--engage_options_path ENGAGE_OPTIONS_PATH] [--makeitso_options_path MAKEITSO_OPTIONS_PATH] [--tiegcm_options_path TIEGCM_OPTIONS_PATH] [--verbose]
Interactive script to prepare a MAGE magnetosphere model run.
options:
-h, --help show this help message and exit
--clobber Overwrite existing options file (default: False).
--debug, -d Print debugging output (default: False).
--mode MODE User mode (BASIC|INTERMEDIATE|EXPERT) (default: BASIC).
--engage_options_path ENGAGE_OPTIONS_PATH, -eo ENGAGE_OPTIONS_PATH
Path to engage JSON file of options (default: None)
--makeitso_options_path MAKEITSO_OPTIONS_PATH, -mo MAKEITSO_OPTIONS_PATH
Path to makeitso JSON file of options (default: None)
--tiegcm_options_path TIEGCM_OPTIONS_PATH, -to TIEGCM_OPTIONS_PATH
Path to tiegcm JSON file of options (default: None)
--verbose, -v Print verbose output (default: False).
The ``--options_path`` option allows the user to specify an existing JSON file
from a previous run of ``engage.py`` so that the entire process of model
generation can be automated. More info on this given below.
The ``--mode`` option specifies the user mode to run in, with ``BASIC`` being the default.
An example in ``BASIC`` mode
----------------------------
This section provdes an annotated example session of ``engage.py`` running
in the default ``BASIC`` mode on the ``derecho`` supercomputer.
1. ``engage`` native parameters will be requested
.. code-block:: bash
engage.py
Name to use for PBS job(s) [geospace]:
Enter an identifying string to use for your model run. This name will be used
as the basis for most of the files created by ``engage.py``, the
``kaiju`` and ``TIE-GCM`` software. The default name is ``geospace``.
.. code-block:: bash
Start date for simulation (yyyy-mm-ddThh:mm:ss) [2016-08-09T09:00:00]:
Stop date for simulation (yyyy-mm-ddThh:mm:ss) [2016-08-09T11:00:00]:
Enter the start and stop date and time for the solar wind data you want to
use. The required data will be fetched from CDAWeb, and converted into a
format usable by the ``kaiju`` software.
.. code-block:: bash
Do you want to split your job into multiple segments? (Y|N) [Y]:
Here ``Y`` is default and is required for the GTR run. This will
split your simulation into multiple PBS jobs that are chained together, with
each using the results of the previous job as a starting point.
.. code-block:: bash
Segment length in simulated seconds [7200.0]: 3600
Enter the length of each segment in simulated seconds. The default is the entire length
of the simulation, but you can enter a shorter time to split the simulation into
multiple segments. For example, if you enter ``3600``, the simulation will be
split into two segments, each one hour long. The first segment will run from
``2016-08-09T09:00:00`` to ``2016-08-09T10:00:00``, and the second segment will run
from ``2016-08-09T10:00:00`` to ``2016-08-09T11:00:00``.
.. code-block:: bash
GAMERA grid type (D|Q|O|H) [Q]:
The codes represent double- (``D``), quad- (``Q``), oct- (``O``) and
hex- (``H``) resolutions in the LFM grid used in the ``kaiju`` software.
.. code-block:: bash
Name of HPC system (derecho|pleiades) [pleiades]: derecho
The ``engage.py`` script supports the ``derecho`` and ``pleiades``
supercomputers. The selection you make here will customize the remaining
prompts for the selected system.
.. code-block:: bash
PBS account name [your_login_name]:
On ``pleiades``, your login name is usable here. On ``derecho``, you will need
a PBS account ID.
.. code-block:: bash
Run directory [.]:
Specify the directory that you wish to perform the simulation in. The
directory will contain all of the files generated by ``engage.py``.
.. code-block:: bash
Path to kaiju installation [YOUR_PATH_HERE]:
Path to kaiju build directory [YOUR_PATH_HERE]:
Enter the paths to the location of your ``kaiju`` code, and the location of
your ``kaiju`` build directory.
.. code-block:: bash
PBS queue name (low|normal|long|debug|devel) [normal]:
Select a PBS queue to use on the selected supercomputer.
.. code-block:: bash
You are responsible for ensuring that the wall time is sufficient
to run a segment of your simulation! Requested wall time for each PBS job
segment (HH:MM:SS) [01:00:00]:
Specify the wall clock time to request for your job (or each segment, if you
split your job into multiple segments).
.. code-block:: bash
Root directory for the simulation [<YOUR_RUN_DIRECTORY_HERE>]:
This is the root directory for your simulation. It will be used to store all
of the files generated by ``engage.py`` and the ``kaiju`` and ``TIE-GCM``
software. The default is the current directory.
.. code-block:: bash
Conda environment to use for the simulation [<YOUR_CONDA_ENVIRONMENT_DIRECTORY_HERE>]:
This is the path to the conda environment that you want to use for the
simulation. This is automatically set to the conda environment that you have
activated when you run the ``engage.py`` script.
2. ``makeitso`` parameters will be requested
.. code-block:: bash
Extend TFIN by dtCouple - 1 seconds (T|F) [T]:
This option allows you to extend the voltron TFIN time by one second. This is
required for coupled runs with TIE-GCM, and is set to ``T`` by default.
.. code-block:: bash
(VOLTRON) Run in GCM mode (T|F) [T]:
This option allows you to run the voltron code in GCM mode, which is required
for coupled runs with TIE-GCM. This is set to ``T`` by default.
.. code-block:: bash
Do you have an existing boundary condition file to use? (Y|N) [N]:
If you already have a file containing solar wind data to use for the inner
boundary conditions of your simulation, enter ``Y``, and you will then be
prompted for the path top the file. If you don't have the file, enter ``N``
and you will be prompted for the date range to use.
.. code-block:: bash
(GAMERA) Relative path to HDF5 file containing solar wind boundary conditions [bcwind.h5]:
This is the path to your existing solar wind file, or the path that
``makeitso.py`` will use to create the file.
.. code-block:: bash
(VOLTRON) File output cadence in simulated seconds [60.0]:
How often (in simulated seconds) the ``kaiju`` software should output results
during the course of the simulation.
The script then runs several additional tools to prepare the files needed for
your simulation.
.. code-block:: bash
Running preprocessing steps.
Generating Quad LFM-style grid ...
Output: lfmQ.h5
Size: (96,96,128)
Inner Radius: 2.000000
Sunward Outer Radius: 30.000000
Tail Outer Radius: 322.511578
Low-lat BC: 45.000000
Ring params:
<ring gid="lfm" doRing="T" Nr="8" Nc1="8" Nc2="16" Nc3="32" Nc4="32" Nc5="64" Nc6="64" Nc7="64" Nc8="64"/>
Writing to lfmQ.h5
Retrieving f10.7 data from CDAWeb
Retrieving solar wind data from CDAWeb
Using Bx fields
Bx Fit Coefficients are [-3.78792744 -0.77915822 -1.0774984 ]
Saving "OMNI_HRO_1MIN.txt_bxFit.png"
Converting to Gamera solar wind file
Found 21 variables and 120 lines
Offsetting from LFM start ( 0.00 min) to Gamera start ( 0.00 min)
Saving "OMNI_HRO_1MIN.txt.png"
Writing Gamera solar wind to bcwind.h5
Reading /glade/derecho/scratch/ewinter/cgs/aplkaiju/kaipy-private/development/kaipy-private/kaipy/rcm/dktable
Reading /glade/derecho/scratch/ewinter/cgs/aplkaiju/kaipy-private/development/kaipy-private/kaipy/rcm/wmutils/chorus_polynomial.txt
Dimension of parameters in Chorus wave model, Kp: 6 MLT: 97 L: 41 Ek: 155
Wrote RCM configuration to rcmconfig.h5
Creating .ini file(s) for run.
Converting .ini file(s) to .xml file(s).
Template creation complete!
Creating PBS job script(s) for run.
The PBS job scripts ['./geospace-00.pbs'] are ready.
The PBS scripts ['./geospace-00.pbs'] have been created, each with a corresponding XML file. To submit the jobs with the proper dependency (to ensure each segment runs in order), please run the script geospace_pbs.sh like this:
bash geospace_pbs.sh
3. ``tiegcmrun`` parameters will be requested
.. code-block:: bash
Instructions:
-> Default Selected input parameter is given in GREEN
-> Warnings and Information are given in YELLOW
-> Errors are given in RED
-> Valid values (if any) are given in brackets eg. (value1 | value2 | value3)
-> Enter '?' for any input parameter to get a detailed description
Run Options:
User Mode = BASIC
Compile = False
Execute = False
Coupling = True
Engage = True
.. code-block:: bash
Directory of model [<YOUR_TIEGCMHOME_HERE>]:
Directory of Tiegcm Data Files [<YOUR_TIEGCMDATA_HERE>]:
This is the path to your TIE-GCM repository and TIE-GCM data directory. This is automatically set to
to the TIEGCMHOME and TIEGCMDATA environment variables
.. code-block:: bash
Standalone Executable [<YOUR_TIEGCM_STANDALONE_EXECUTABLE_HERE>]:
This is the path to the TIE-GCM standalone executable. This is automatically set
to the ``tiegcm.exe`` in current directory.
.. code-block:: bash
Coupled Executable [<YOUR_TIEGCM_COUPLED_EXECUTABLE_HERE>]:
This is the path to the TIE-GCM coupled executable. This is automatically set
to the ``tiegcm.x`` in current directory.
.. code-block:: bash
Low = 70, Medium = 140 , High = 200
F107 flux level for TIEGCM spin up (low|medium|high) [low]:
This is the F10.7 flux level to use for the TIE-GCM source file in spin-up period. The
default is ``low``, which corresponds to a value of 70. The other options are
``medium`` (140) and ``high`` (200).
.. code-block:: bash
SOURCE file location [/glade/campaign/hao/itmodel/tiegcm3.0/new_data/source/junsol_f70.nc]:
This is the path to the TIE-GCM source file to use for the spin-up period. The default is
automatically selected based on the start date of your simulation.
.. code-block:: bash
Selected date in source file Example: (173,0,0,0) [173 0 0 0]:
STEP number [30]:
NSTEP_SUB number [10]:
These parameters are set as default by the ``tiegcmrun``
.. code-block:: bash
Secondary Output Fields [['TN', 'UN', 'VN', 'NE', 'TEC', 'POTEN', 'Z', 'ZG']] / ENTER to go next:
These are the secondary output fields to include in the TIE-GCM output.
The default is a set of fields that are commonly used in geospace simulations.
You can add another filed if you wish, or just hit :kbd:`Return` to accept the default.
.. code-block:: bash
High-latitude potential model that is going to be used (HEELIS|WEIMER) [HEELIS]:
This is the high-latitude potential model to use in the TIE-GCM simulation.
The default is ``HEELIS``, which is the Heelis potential model is required for
coupled runs with the ``kaiju`` software.
.. code-block:: bash
If GPI_NCFILE is specified, then KP and POWER/CTPOTEN are skipped. If further POTENTIAL_MODEL is WEIMER and IMF_NCFILE is specified, then the Weimer model and aurora will be driven by the IMF data, and only F107 and F107A will be read from the GPI data file.
GPI file [/glade/campaign/hao/itmodel/tiegcm3.0/new_data/boundary_files/GPI/gpi_1960001-2024332.nc]:
This is the path to the GPI file to use for the TIE-GCM simulation which contrains solar wind
data. The default is automatically selected based on the start date of your simulation.
After these inputs, the script interpolates source file for TIEGCM, and generates XML and
PBS files for the run, as well as a grid file for use in the model.
You should see output similar to this:
.. code-block:: bash
/glade/derecho/scratch/nikhilr/GTR58 exitsts
/glade/derecho/scratch/nikhilr/GTR58 exitsts
/glade/derecho/scratch/nikhilr/GTR58 exitsts
Interpolating primary file /glade/campaign/hao/itmodel/tiegcm3.0/new_data/source/junsol_f70.nc to create new primary file /glade/derecho/scratch/nikhilr/GTR58/tiegcm_standalone/geospace-tiegcm-standalone_prim.nc at horizontal resolution 2.5 and vertical resolution 0.25 with zitop 7.0.
Creating new primary file: /glade/derecho/scratch/nikhilr/GTR58/tiegcm_standalone/geospace-tiegcm-standalone_prim.nc
pbs_scripts = ['./geospace-01.pbs', './geospace-02.pbs']
submit_all_jobs_script = geospace_pbs.sh
When finished, the script creates the file ``runid.json``, where ``runid`` is
the identifying string for your simulation. This file contains a record of all
of the parameters used in your simulation. This file can be passed back to
``engage.py`` in a subsequent session to repeat the simulation, and also
provides a convenient starting point for minor tweaks to your simulation
parameters.
There are several types files created for each of the jobs, including:
* ``*.pbs``
These are the PBS scripts that will be submitted to the job scheduler to run
the segments of the simulation.
* ``*.xml``
These are the XML files that contain the parameters for GAMERA and RAIJU of the
segment.
* ``*.inp``
These are the namelist files that contain parameters for TIE-GCM of the segment.
* ``*.json``
These are the JSON files that contain the parameters for the simulation. They
are generated by the ``engage.py`` script with all the parameters required to run the
simulation.
The run is divided into segments:
* ``geospace-SPINUP.*``
This segment runs the GAMERA model to create the initial conditions for the
simulation. It is run first, and its output is used by the next segment.
* ``geospace-WARMUP-**.*``
These segments runs the GAMERA RAIJU model to "warm up" for for the coupled model execution.
The ``-01``, ``-02``, etc. suffixes indicate the segment number, and the
segments are run in order.
* ``tiegcm_standalone-**.*``
This segment runs the TIE-GCM model to create the initial conditions for the coupled model.
The ``-01`` to ``-02``, etc. suffixes indicate the segment number, and the
segments are run in order.
* ``geospace-**.*``
These segments runs the GTR coupled modele. The ``-01``, ``-02``, etc.
suffixes indicate the segment number, and the segments are run
in order.
This image shows how the segments are run in order:
.. image:: ../running/GTRSegment.png
Additional parameters in ``INTERMEDIATE`` and ``EXPERT`` mode
-------------------------------------------------------------
Many more parameters are available in ``INTERMEDIATE`` and ``EXPERT`` modes.
These parameters are documented in the file ``option_descriptions.json``,
which is stored in the same directory as the ``engage.py`` script.
Using JSON files for ``engage.py``
---------------------------------
The ``engage.py`` script can also be run in a non-interactive mode, where it
reads a JSON file containing the parameters for the simulation. This allows
you to automate the process of running the simulation, and to easily repeat
the simulation with the same parameters.
The ``engage.py`` script requires three JSON files to be specified:
* ``engage_options_path``
This is the path to the JSON file containing the parameters for the
``engage.py`` script. It contains the parameters that are specific to the
``engage.py`` script, such as the run ID, start and stop dates, and so on.
* ``makeitso_options_path``
This is the path to the JSON file containing the parameters for the
``makeitso.py`` script. It contains the parameters that are specific to the
``makeitso.py`` script, such as the GAMERA grid type, segment length, and so on.
* ``tiegcm_options_path``
This is the path to the JSON file containing the parameters for the
``tiegcmrun`` script. It contains the parameters that are specific to the
TIE-GCM simulation, such as the source file, F10.7 flux level, and so on.
To run the ``engage.py`` script in non-interactive mode, you can use the
following command:
.. code-block:: bash
engage.py --engage_options_path /path/to/engage_input.json --makeitso_options_path /path/to/makeitso_input.json --tiegcm_options_path /path/to/tiegcm_input.json
Here are templates for the JSON files:
- Derecho:
- :download:`engage_input.json <engage_template/derecho/engage_input.json>`
- :download:`makeitso_input.json <engage_template/derecho/makeitso_input.json>`
- :download:`tiegcm_input.json <engage_template/derecho/tiegcmrun_input.json>`
- Pleiades:
- :download:`engage_input.json <engage_template/pleiades/engage_input.json>`
- :download:`makeitso_input.json <engage_template/pleiades/makeitso_input.json>`
- :download:`tiegcm_input.json <engage_template/pleiades/tiegcmrun_input.json>`
These JSON files can be used as a starting point for your own simulations. You will
need to modify certain parameters in them:
- engage_input.json:
- start_date: The start date of your simulation.
- stop_date: The stop date of your simulation.
- segment_duration: The duration of each segment in simulated seconds.
- gamera_grid_type: The GAMERA grid type to use (D, Q, O, or H).
- kaiju_install_directory: The path to your ``kaiju`` installation directory.
- kaiju_build_directory: The path to your ``kaiju`` build directory.
- makeitso_input.json:
- Automcatically generated by the ``engage.py`` script, but you can modify the
parameters if needed.
- tiegcm_input.json:
- modeldir: The path to your TIE-GCM repository.
- tgcmdata: The path to your TIE-GCM data directory.
- modelexe: The path to the TIE-GCM standalone executable.
- coupled_modelexe: The path to the TIE-GCM coupled executable.
- solar_flux_level: The F10.7 flux level to use for the TIE-GCM source file in spin-up period (low, medium, or high).
- SECFLDS: The secondary output fields to include in the TIE-GCM output.
- Automcatically generated by the ``engage.py`` script, but you can modify the
parameters if needed.

28
docs/source/makeitso/engage_template/derecho/engage_input.json Normal file
View File

@@ -0,0 +1,28 @@
{
"simulation": {
"job_name": "geospace",
"start_date": "2016-08-09T09:00:00",
"stop_date": "2016-08-09T11:00:00",
"use_segments": "Y",
"segment_duration": "3600",
"gamera_grid_type": "D",
"hpc_system": "derecho"
},
"pbs": {
"account_name": "P28100045",
"run_directory": ".",
"kaiju_install_directory": "/glade/u/home/nikhilr/kaiju_engage/kaiju-private",
"kaiju_build_directory": "/glade/u/home/nikhilr/kaiju_engage/kaiju-private/build_mpi",
"queue": "main",
"job_priority": "premium",
"walltime": "03:00:00"
},
"coupling": {
"gr_warm_up_time": 14400,
"gcm_spin_up_time": 604800,
"root_directory": "/glade/derecho/scratch/nikhilr/GTR34",
"conda_env": "/glade/work/nikhilr/conda-envs/kaiju",
"tfin_delta": "T",
"doGCM": "T"
}
}

139
docs/source/makeitso/engage_template/derecho/makeitso_input.json Normal file
View File

@@ -0,0 +1,139 @@
{
"simulation": {
"bcwind_available": "N",
"gamera_grid_inner_radius": "2.0",
"gamera_grid_outer_radius": "30.0"
},
"pbs": {
},
"gamera": {
"sim": {
"doH5g": "T",
"icType": "user",
"runid": "gtr34",
"pdmb": "0.75",
"rmeth": "8C"
},
"floors": {
"dFloor": "1.0e-4",
"pFloor": "1.0e-6"
},
"timestep": {
"doCPR": "T",
"limCPR": "0.20"
},
"restart": {
"doRes": "F",
"nRes": "-1",
"resID": "gtr34"
},
"physics": {
"doBoris": "T",
"Ca": "10.0",
"doMHD": "T"
},
"ring": {
"doRing": "T",
"gid": "lfm"
},
"wind": {
"tsfile": "bcwind.h5"
},
"source": {
"doSource": "T",
"doBounceDT": "T",
"nBounce": "1.0",
"doWolfLim": "T"
},
"iPdir": {
"bcPeriodic": "F"
},
"jPdir": {
"bcPeriodic": "F"
},
"kPdir": {
"bcPeriodic": "T"
},
"coupling": {
"blockHalo": "T"
}
},
"voltron": {
"time": {
},
"spinup": {
"doSpin": "T",
"tSpin": "7200.0",
"tIO": "0.0"
},
"output": {
"dtOut": "60.0",
"tsOut": "300.0"
},
"coupling": {
"doQkSquish": "T",
"doGCM": "F",
"qkSquishStride": "2",
"dtCouple": "5.0",
"doDeep": "T",
"imType": "RCM",
"doAsyncCoupling": "F"
},
"restart": {
"dtRes": "1800.0"
},
"imag": {
"doInit": "T"
},
"helpers": {
"doSquishHelp": "T"
}
},
"chimp": {
"units": {
"uid": "EARTHCODE"
},
"fields": {
"grType": "lfm"
},
"domain": {
"dtype": "MAGE"
},
"tracer": {
"epsds": "0.05"
}
},
"remix": {
"conductance": {
"doStarlight": "T",
"apply_cap": "T",
"const_sigma": "F"
},
"precipitation": {
"aurora_model_type": "LINMRG",
"beta": "0.2",
"doAuroralSmooth": "F"
}
},
"rcm": {
"rcmdomain": {
"domType": "ELLIPSE"
},
"ellipse": {
"xSun": "12.5",
"yDD": "15.0",
"xTail": "-15.0",
"isDynamic": "T"
},
"grid": {
"LowLat": "30.0",
"HiLat": "75.0"
},
"plasmasphere": {
"isDynamic": "T",
"initKp": "5",
"doRefill": "T",
"tAvg": "60.0"
}
}
}

111
docs/source/makeitso/engage_template/derecho/tiegcmrun_input.json Normal file
View File

@@ -0,0 +1,111 @@
{
"simulation": {
},
"model": {
"data": {
"modeldir": "/glade/u/home/nikhilr/kaiju_engage/tiegcm",
"tgcmdata": "/glade/campaign/hao/itmodel/tiegcm3.0/new_data",
"modelexe": "/glade/derecho/scratch/nikhilr/GTR57/tiegcm.exe",
"coupled_modelexe": "/glade/derecho/scratch/nikhilr/GTR57/tiegcm.x"
},
"specification": {
"zitop": 7
}
},
"inp": {
"LABEL": "tiegcm",
"CALENDAR_ADVANCE": 1,
"NSTEP_SUB": "10",
"solar_flux_level": "low",
"SECFLDS": [
"TN",
"UN",
"VN",
"NE",
"TEC",
"POTEN",
"Z",
"ZG"
],
"ELECTRON_HEATING": 6,
"POTENTIAL_MODEL": "HEELIS",
"GPI_NCFILE": "/glade/campaign/hao/itmodel/tiegcm3.0/new_data/boundary_files/GPI/gpi_1960001-2025120.nc",
"IMF_NCFILE": null,
"KP": null,
"POWER": null,
"CTPOTEN": null,
"BXIMF": null,
"BYIMF": null,
"BZIMF": null,
"SWDEN": null,
"SWVEL": null,
"F107": null,
"F107A": null,
"ONEWAY": false,
"AMIENH": null,
"AMIESH": null,
"AURORA": null,
"DYNAMO": null,
"CALC_HELIUM": null,
"EDDY_DIF": null,
"JOULEFAC": null,
"COLFAC": null,
"OPDIFFCAP": "2e9",
"OPDIFFRATE": "0.3",
"OPDIFFLEV": "7",
"OPFLOOR": "3000",
"OPRATE": "0.3",
"OPLEV": "7",
"OPLATWIDTH": "20",
"TE_CAP": "8000",
"TI_CAP": "8000",
"CURRENT_PG": null,
"CURRENT_KQ": null,
"ET": null,
"SAPS": null,
"DOECLIPSE": null,
"ECLIPSE_LIST": null,
"HE_COEFS_NCFILE": "/glade/campaign/hao/itmodel/tiegcm3.0/new_data/boundary_files/other/he_coefs_dres.nc",
"BGRDDATA_NCFILE": null,
"CTMT_NCFILE": null,
"SABER_NCFILE": null,
"TIDI_NCFILE": null,
"TIDE": null,
"TIDE2": null,
"MIXFILE": null,
"NUDGE_NCPRE": null,
"NUDGE_NCPOST": null,
"NUDGE_NCFILE": null,
"NUDGE_FLDS": null,
"NUDGE_LBC": null,
"NUDGE_F4D": null,
"NUDGE_USE_REFDATE": null,
"NUDGE_REFDATE": null,
"NUDGE_SPONGE": null,
"NUDGE_DELTA": null,
"NUDGE_POWER": null,
"NUDGE_ALPHA": null,
"other_input": [
null
]
},
"job": {
"modules": [
"ncarenv/23.09",
"cmake/3.26.3",
"craype/2.7.31",
"intel-classic/2023.2.1",
"cray-mpich/8.1.27",
"ncarcompilers/1.0.0",
"mkl/2023.2.0",
"hdf5-mpi/1.12.2",
"netcdf-mpi/4.9.2",
"esmf/8.6.0",
"conda"
],
"other_job": [
null
]
}
}

27
docs/source/makeitso/engage_template/pleiades/engage_input.json Normal file
View File

@@ -0,0 +1,27 @@
{
"simulation": {
"job_name": "geospace",
"start_date": "2016-08-09T09:00:00",
"stop_date": "2016-08-09T11:00:00",
"use_segments": "Y",
"segment_duration": "3600",
"gamera_grid_type": "D",
"hpc_system": "pleiades"
},
"pbs": {
"account_name": "nrao3",
"run_directory": ".",
"kaiju_install_directory": "/nobackupp27/nrao3/engage_build/kaiju-private",
"kaiju_build_directory": "/nobackupp27/nrao3/engage_build/kaiju-private/build_mpi",
"group_list": "None",
"queue": "normal",
"walltime": "03:00:00"
},
"coupling": {
"gr_warm_up_time": 14400,
"gcm_spin_up_time": 604800,
"root_directory": "/nobackupp27/nrao3/GTR152",
"tfin_delta": "T",
"doGCM": "T"
}
}

140
docs/source/makeitso/engage_template/pleiades/makeitso_input.json Normal file
View File

@@ -0,0 +1,140 @@
{
"simulation": {
"bcwind_available": "N",
"segment_duration": "3600",
"gamera_grid_inner_radius": "2.0",
"gamera_grid_outer_radius": "30.0"
},
"pbs": {
},
"gamera": {
"sim": {
"doH5g": "T",
"icType": "user",
"pdmb": "0.75",
"rmeth": "8C"
},
"floors": {
"dFloor": "1.0e-4",
"pFloor": "1.0e-6"
},
"timestep": {
"doCPR": "T",
"limCPR": "0.20"
},
"restart": {
"doRes": "F",
"nRes": "-1",
"resID": "GTR152D"
},
"physics": {
"doBoris": "T",
"Ca": "10.0",
"doMHD": "T"
},
"ring": {
"doRing": "T",
"gid": "lfm"
},
"wind": {
"tsfile": "bcwind.h5"
},
"source": {
"doSource": "T",
"doBounceDT": "T",
"nBounce": "1.0",
"doWolfLim": "T"
},
"iPdir": {
"bcPeriodic": "F"
},
"jPdir": {
"bcPeriodic": "F"
},
"kPdir": {
"bcPeriodic": "T"
},
"coupling": {
"blockHalo": "F"
}
},
"voltron": {
"time": {
"tFin": 7200.0
},
"spinup": {
"doSpin": "T",
"tSpin": "7200.0",
"tIO": "0.0"
},
"output": {
"dtOut": "60.0",
"tsOut": "300.0"
},
"coupling": {
"doQkSquish": "T",
"doGCM": "F",
"qkSquishStride": "2",
"dtCouple": "5.0",
"doDeep": "T",
"imType": "RCM",
"doAsyncCoupling": "T"
},
"restart": {
"dtRes": "1800.0"
},
"imag": {
"doInit": "T"
},
"helpers": {
"doSquishHelp": "T"
}
},
"chimp": {
"units": {
"uid": "EARTHCODE"
},
"fields": {
"grType": "lfm"
},
"domain": {
"dtype": "MAGE"
},
"tracer": {
"epsds": "0.05"
}
},
"remix": {
"conductance": {
"doStarlight": "T",
"apply_cap": "T",
"const_sigma": "F"
},
"precipitation": {
"aurora_model_type": "LINMRG",
"beta": "0.2",
"doAuroralSmooth": "F"
}
},
"rcm": {
"rcmdomain": {
"domType": "ELLIPSE"
},
"ellipse": {
"xSun": "12.5",
"yDD": "15.0",
"xTail": "-15.0",
"isDynamic": "T"
},
"grid": {
"LowLat": "30.0",
"HiLat": "75.0"
},
"plasmasphere": {
"isDynamic": "T",
"initKp": "5",
"doRefill": "T",
"tAvg": "60.0"
}
}
}

96
docs/source/makeitso/engage_template/pleiades/tiegcmrun_input.json Normal file
View File

@@ -0,0 +1,96 @@
{
"simulation": {
"job_name": "GTR152D",
"hpc_system": "pleiades"
},
"model": {
"data": {
"modeldir": "/nobackupp27/nrao3/engage_build/tiegcm",
"tgcmdata": "/nobackup/nrao3/tiegcm/tiegcm3.0/data",
"modelexe": "/nobackupp27/nrao3/GTR156/tiegcm.exe",
"coupled_modelexe": "/nobackupp27/nrao3/GTR156/tiegcm.x"
},
"specification": {
"zitop": 7
}
},
"inp": {
"LABEL": "tiegcm",
"CALENDAR_ADVANCE": 1,
"NSTEP_SUB": "10",
"solar_flux_level": "low",
"SECFLDS": [
"TN",
"UN",
"VN",
"NE",
"TEC",
"POTEN",
"Z",
"ZG"
],
"ELECTRON_HEATING": 6,
"POTENTIAL_MODEL": "HEELIS",
"GPI_NCFILE": "/nobackup/nrao3/tiegcm/tiegcm3.0/data/gpi_mgii_1979001-2010212.nc",
"IMF_NCFILE": null,
"KP": null,
"POWER": null,
"CTPOTEN": null,
"BXIMF": null,
"BYIMF": null,
"BZIMF": null,
"SWDEN": null,
"SWVEL": null,
"F107": null,
"F107A": null,
"ONEWAY": false,
"AMIENH": null,
"AMIESH": null,
"AURORA": null,
"DYNAMO": null,
"CALC_HELIUM": null,
"EDDY_DIF": null,
"JOULEFAC": null,
"COLFAC": null,
"OPDIFFCAP": "2e9",
"OPDIFFRATE": "0.3",
"OPDIFFLEV": "7",
"OPFLOOR": "3000",
"OPRATE": "0.3",
"OPLEV": "7",
"OPLATWIDTH": "20",
"TE_CAP": "8000",
"TI_CAP": "8000",
"CURRENT_PG": null,
"CURRENT_KQ": null,
"ET": null,
"SAPS": null,
"DOECLIPSE": null,
"ECLIPSE_LIST": null,
"HE_COEFS_NCFILE": "/nobackup/nrao3/tiegcm/tiegcm3.0/data/he_coefs_dres.nc",
"BGRDDATA_NCFILE": null,
"CTMT_NCFILE": null,
"SABER_NCFILE": null,
"TIDI_NCFILE": null,
"TIDE": null,
"TIDE2": null,
"MIXFILE": null,
"NUDGE_NCPRE": null,
"NUDGE_NCPOST": null,
"NUDGE_NCFILE": null,
"NUDGE_FLDS": null,
"NUDGE_LBC": null,
"NUDGE_F4D": null,
"NUDGE_USE_REFDATE": null,
"NUDGE_REFDATE": null,
"NUDGE_SPONGE": null,
"NUDGE_DELTA": null,
"NUDGE_POWER": null,
"NUDGE_ALPHA": null,
"other_input": [
null
]
},
"job": {
}
}

1
docs/source/makeitso/index.rst
View File

@@ -10,4 +10,5 @@ and running MAGE and GAMERA-helio applications.
:maxdepth: 1
makeitso
engage
makeitso-gamhelio

35
docs/source/python/derecho.rst
View File

@@ -13,37 +13,28 @@ packages, and packages are only taken from PyPI.
These instructions assume you are using the ``bash`` shell.
.. note:: These instructions are designed for novice users. If you are
comfortable with building and managing ``conda``-based environments, feel
free to build your own environment using the CISL-provided
`conda <https://ncar-hpc-docs.readthedocs.io/en/stable/environment-and-software/user-environment/conda>`_
software.
Building the python environment
-------------------------------
To create a Python environment for ``kaiju``, we first install Python, then
create the environment, then populate it with the required Python packages.
To create a Python environment for ``kaiju``, we first install/load the
Conda software package, then create the environment, then populate it with
the required Python packages.
.. note::
These instructions are designed for novice users. If you are
comfortable with building and managing ``conda``-based environments, feel
free to look into the CISL-provided
`conda documentation <https://ncar-hpc-docs.readthedocs.io/en/stable/environment-and-software/user-environment/conda>`_
for futher details.
.. code-block:: bash
# Download the Miniconda installer.
cd $HOME
wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86.sh
# Run the installer.
# Install into $HOME/miniconda3, use all defaults.
bash ./Miniconda3-latest-Linux-x86.sh
# Make sure the shell is properly configured.
# Replace with rc file for your shell.
source $HOME/.bashrc
# Update everything to latest version.
conda activate base
conda update --all
# Load the conda module.
module load conda
# Now create the environment for kaiju, specifying only the python
# version.

33
docs/source/python/pleiades.rst
View File

@@ -13,6 +13,15 @@ packages, and packages are only taken from PyPI.
These instructions assume you are using the ``bash`` shell.
Building the python environment
-------------------------------
To create a Python environment for ``kaiju``, we first install/load the
Conda software package, then create the environment, then populate it with
the required Python packages.
.. note:: These instructions are designed for novice users. If you are
comfortable with building and managing ``conda``-based environments, feel
free to build your own environment using the NAS-provided
@@ -20,30 +29,12 @@ These instructions assume you are using the ``bash`` shell.
software.
Building the python environment
-------------------------------
To create a Python environment for ``kaiju``, we first install Python, then
create the environment, then populate it with the required Python packages.
.. code-block:: bash
# Download the Miniconda installer.
cd $HOME
wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86.sh
# Run the installer.
# Install into $HOME/miniconda3, use all defaults.
bash ./Miniconda3-latest-Linux-x86.sh
# Make sure the shell is properly configured.
# Replace with rc file for your shell.
source $HOME/.bashrc
# Update everything to latest version.
conda activate base
conda update --all
# Load the miniconda3 module.
module use -a /swbuild/analytix/tools/modulefiles
module load miniconda3/v4
# Now create the environment for kaiju, specifying only the python
# version.

BIN
docs/source/running/GTRSegment.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 42 KiB

4
docs/source/running/geoQuickStart.rst → docs/source/running/geoGRQuickStart.rst
View File

@@ -1,4 +1,4 @@
Geospace Quick Start
Geospace - Gamera Raiju (GR) Quick Start
====================
@@ -14,7 +14,7 @@ Before you begin
.. code-block:: bash
source /path/to/your/kaiju-clone/scripts/setupEnvironment.sh
source /path/to/your/kaiju/scripts/setupEnvironment.sh
source /path/to/your/kaipy/scripts/setupEnvironment.sh

483
docs/source/running/geoGTRQuickStart.rst Normal file
View File

@@ -0,0 +1,483 @@
Geospace - Gamera TIE-GCM Raiju (GTR) Quick Start
====================
These instructions illustrate the process of running a magnetosphere
simulation using the MAGE model in the ``kaiju`` code.
Before you begin
----------------
*Source* (not *run*) the environment setup scripts for the ``kaiju`` and
``kaipy`` software. For example:
.. code-block:: bash
source /path/to/your/kaiju-clone/scripts/setupEnvironment.sh
source /path/to/your/kaipy/scripts/setupEnvironment.sh
export TIEGCMHOME=/path/to/your/tiegcm
export TIEGCMDATA=/path/to/your/tiegcm/data
.. note::
The ``TIEGCMHOME`` and ``TIEGCMDATA`` environment variables are required
for running the GTR model. They should point to the TIE-GCM source code
directory and the TIE-GCM data directory, respectively.
The TIEGCMDATA directory is located in the following locations:
- On ``derecho``: ``/glade/campaign/hao/itmodel/tiegcm3.0/new_data``
- On ``pleiades``: ``/nobackup/nrao3/tiegcm/tiegcm3.0/data``
Build guide for TIE-GCM
***********************
`TIE-GCM <https://tiegcm-docs.readthedocs.io/>`_. is a comprehensive, first-principles, three-dimensional,
non-linear representation of the coupled thermosphere and ionosphere system that includes a self-consistent solution
of the middle and low-latitude dynamo field.
Two TIE-GCM executables are required for running the GTR model:
- TIEGCM Standalone: This is the TIE-GCM code that runs independently and is used for initialization of the model.
- TIEGCM Coupled: This is the TIE-GCM code that runs in a coupled mode with the GR model, providing
real-time updates to the thermosphere and ionosphere conditions during the simulation.
Depending on the Gamera resolution you will need to compile different TIEGCM resolution executables:
- For a ``D`` run
- TIEGCM Standalone: horires = 2.5, vertres = 0.25(1/4), mres = 2
- TIEGCM Coupled: horires = 2.5, vertres = 0.25(1/4), mres = 2
- For a ``Q`` run
- TIEGCM Standalone: horires = 2.5, vertres = 0.25(1/4), mres = 2
- TIEGCM Coupled: horires = 1.25, vertres = 0.125(1/8), mres = 1
- For a ``O`` run
- TIEGCM Standalone: horires = 1.25, vertres = 0.125(1/8), mres = 1
- TIEGCM Coupled: horires = 0.625, vertres = 0.0625(1/16), mres = 0.5
The TIE-GCM code is built using the ``tiegcmrun`` script, which is provided in
the ``tiegcm`` code repository. The script is provided in the
``tiegcm/tiegcmrun`` directory. More information on ``tiegcmrun.py`` can be found
in the `TIEGCM Quick Start Guide <https://tiegcm-docs.readthedocs.io/en/latest/tiegcm/quickstart.html>`_.
.. important::
Make sure to load the modules lised in the ``kaiju`` build instructions
before running the ``tiegcmrun`` script. (:doc:`Derecho </building/buildDerecho>` or :doc:`Pleiades </building/buildPleiades>`)
Lets take an example of building the TIE-GCM code for a ``Q`` run on ``derecho``:
####################
1. First we will create a directory for the TIE-GCM build
.. code-block:: bash
mkdir tiegcm_build
cd tiegcm_build
2. Next, we will build the standalone TIE-GCM executable by running the ``tiegcmrun.py`` script with the ``-oc``.
.. note::
The ``-oc`` option stands for "only compile", which means that the script will only compile the code and not run it.
Since the Gamera resolution is ``Q``, we will set the horizontal resolution for the standalone TIE-GCM to 2.5 degrees,
vertical resolution to 0.25 degrees, and the top altitude to 7.0 RE.
.. code-block:: bash
$TIEGCMHOME/tiegcmrun/tiegcmrun.py -oc
Instructions:
-> Default Selected input parameter is given in GREEN
-> Warnings and Information are given in YELLOW
-> Errors are given in RED
-> Valid values (if any) are given in brackets eg. (value1 | value2 | value3)
-> Enter '?' for any input parameter to get a detailed description
Run Options:
User Mode = BASIC
Compile = True
Execute = False
Coupling = False
Name of HPC system (derecho|pleiades|linux) [derecho]:
Standalone Executable [/glade/derecho/scratch/nikhilr/tiegcm_build/exec/tiegcm.exe]:
Horizontal Resolution (Deg) (5.0|2.5|1.25|0.625) [2.5]:
Vertical Resolution (Scale Height) (1/2|1/4|1/8|1/16) [1/4]:
Magnetic grid resolution (Degree) (2|1|0.5) [2]:
After these inputs, the script will compile the TIE-GCM code and create the standalone executable and should output something like this:
.. code-block:: bash
..
..
gmake[1]: Leaving directory '/glade/derecho/scratch/nikhilr/tiegcm_build/exec'
Executable copied from /glade/derecho/scratch/nikhilr/tiegcm_build/exec/tiegcm.exe to /glade/derecho/scratch/nikhilr/tiegcm_build/stdout
3. Next, we will build the coupled TIE-GCM executable by running the ``tiegcmrun.py`` script with the ``-oc`` and ``-co`` options.
.. note::
The ``-co`` option stands for "coupled", which means that the script will compile the code for the coupled TIE-GCM executable.
Since the Gamera resolution is ``Q``, we will set the horizontal resolution for the coupled TIE-GCM to 1.25 degrees,
vertical resolution to 0.125 degree.
.. code-block:: bash
$TIEGCMHOME/tiegcmrun/tiegcmrun.py -oc -co
Instructions:
-> Default Selected input parameter is given in GREEN
-> Warnings and Information are given in YELLOW
-> Errors are given in RED
-> Valid values (if any) are given in brackets eg. (value1 | value2 | value3)
-> Enter '?' for any input parameter to get a detailed description
Run Options:
User Mode = BASIC
Compile = True
Execute = False
Coupling = True
Name of HPC system (derecho|pleiades|linux) [derecho]:
Coupled Executable [/glade/derecho/scratch/nikhilr/tiegcm_build/exec/tiegcm.x]:
Horizontal Resolution (Deg) (5.0|2.5|1.25|0.625) [2.5]: 1.25
Vertical Resolution (Scale Height) (1/2|1/4|1/8|1/16) [1/8]:
Magnetic grid resolution (Degree) (2|1|0.5) [1]:
After these inputs, the script will compile the TIE-GCM code and create the coupled executable and should output something like this:
.. code-block:: bash
..
..
gmake[1]: Leaving directory '/glade/derecho/scratch/nikhilr/tiegcm_build/exec'
Executable copied from /glade/derecho/scratch/nikhilr/tiegcm_build/exec/tiegcm.x to /glade/derecho/scratch/nikhilr/tiegcm_build/stdout
4. You should now see the following files in your run directory:
.. code-block:: bash
ls
exec hist stdout
The executables are located in the ``stdout`` directory, and the stdout files
.. code-block:: bash
ls stdout
defs.h tiegcm.exe tiegcm.x
Running a magnetosphere simulation with MAGE
--------------------------------------------
The MAGE software needs several files in order to run. The detailed steps
for creating these files have been combined into a script called
``engage.py``. The script is provided in the ``kaiju`` code repository. More
information on ``engage.py`` is available
:doc:`here </makeitso/engage>`.
You can see the options supported by ``engage.py`` by running it with the
``--help`` or ``-h`` command-line option.
.. code-block:: bash
engage.py --help
usage: engage.py [-h] [--clobber] [--debug] [--mode MODE] [--engage_options_path ENGAGE_OPTIONS_PATH] [--makeitso_options_path MAKEITSO_OPTIONS_PATH] [--tiegcm_options_path TIEGCM_OPTIONS_PATH] [--verbose]
Interactive script to prepare a MAGE magnetosphere model run.
options:
-h, --help show this help message and exit
--clobber Overwrite existing options file (default: False).
--debug, -d Print debugging output (default: False).
--mode MODE User mode (BASIC|INTERMEDIATE|EXPERT) (default: BASIC).
--engage_options_path ENGAGE_OPTIONS_PATH, -eo ENGAGE_OPTIONS_PATH
Path to engage JSON file of options (default: None)
--makeitso_options_path MAKEITSO_OPTIONS_PATH, -mo MAKEITSO_OPTIONS_PATH
Path to makeitso JSON file of options (default: None)
--tiegcm_options_path TIEGCM_OPTIONS_PATH, -to TIEGCM_OPTIONS_PATH
Path to tiegcm JSON file of options (default: None)
--verbose, -v Print verbose output (default: False).
For this example, we will run the code on ``derecho``, and use the default
``BASIC`` mode, which requires the minimum amount of input from the user. At
each prompt, you can either type in a value, or hit the :kbd:`Return` key to
accept the default value (shown in square brackets at the end of the prompt).
Copy the executables you built in the previous steps to your run directory.
.. code-block:: bash
cp tiegcm_build/stdout/tiegcm.exe .
cp tiegcm_build/stdouttiegcm.x .
cp $KAIJUHOME/build_mpi/bin/voltron_mpi.x .
To get started, run ``engage.py`` with no arguments:
.. code-block:: bash
$KAIJUHOME/scripts/makeitso/engage.py
tiegcmrum from /glade/u/home/nikhilr/kaiju_engage/tiegcm/tiegcmrun/tiegcmrun.py
makeitso from /glade/u/home/nikhilr/kaiju_engage/kaiju-private/scripts/makeitso/makeitso.py
Name to use for PBS job(s) [geospace]:
Start date for simulation (yyyy-mm-ddThh:mm:ss) [2016-08-09T09:00:00]:
Stop date for simulation (yyyy-mm-ddThh:mm:ss) [2016-08-09T11:00:00]:
Do you want to split your job into multiple segments? (Y|N) [Y]:
Segment length in simulated seconds [7200.0]: 3600
GAMERA grid type (D|Q|O|H) [Q]:
Name of HPC system (derecho|pleiades) [pleiades]: derecho
PBS account name [<YOUR_ACCOUNT_HERE>]:
Run directory [.]:
Path to kaiju installation [<YOUR_KAIJUHOME_HERE>]:
Path to kaiju build directory [<YOUR_BUILD_DIRECTORY_HERE>]:
PBS queue name (develop|main) [main]:
Job priority (regular|economy) [economy]:
WARNING: You are responsible for ensuring that the wall time is sufficient to run a segment of your simulation!
Requested wall time for each PBS job segment (HH:MM:SS) [01:00:00]: 12:00:00
Root directory for the simulation [<YOUR_RUN_DIRECTORY_HERE>]:
Conda environment to use for the simulation [<YOUR_CONDA_ENVIRONMENT_DIRECTORY_HERE>]:
``engage.py`` will then prompt you for the following additional information from ``makeitso``:
.. code-block:: bash
Extend TFIN by dtCouple - 1 seconds (T|F) [T]:
(VOLTRON) Run in GCM mode (T|F) [T]:
Do you have an existing boundary condition file to use? (Y|N) [N]:
(GAMERA) Relative path to HDF5 file containing solar wind boundary conditions [bcwind.h5]:
(VOLTRON) File output cadence in simulated seconds [60.0]:
After these inputs, the script fetches data from CDAWeb for the specified time
range to use in the solar wind boundary condition file.
You should see output similar to this:
.. code-block:: bash
GGenerating Quad LFM-style grid ...
Output: lfmQ.h5
Size: (96,96,128)
Inner Radius: 2.000000
Sunward Outer Radius: 30.000000
Tail Outer Radius: 322.511578
Low-lat BC: 45.000000
Ring params:
<ring gid="lfm" doRing="T" Nr="8" Nc1="8" Nc2="16" Nc3="32" Nc4="32" Nc5="64" Nc6="64" Nc7="64" Nc8="64"/>
Writing to lfmQ.h5
14-Jun-25 19:30:03: /glade/work/nikhilr/conda-envs/kaiju-3.12/lib/python3.12/site-packages/spacepy/time.py:2448: UserWarning: Leapseconds may be out of date. Use spacepy.toolbox.update(leapsecs=True)
_read_leaps()
Retrieving f10.7 data from CDAWeb
Retrieving solar wind data from CDAWeb
Using Bx fields
Bx Fit Coefficients are [-3.78792744 -0.77915822 -1.0774984 ]
Saving "OMNI_HRO_1MIN.txt_bxFit.png"
Converting to Gamera solar wind file
Found 21 variables and 120 lines
Offsetting from LFM start ( 0.00 min) to Gamera start ( 0.00 min)
Saving "OMNI_HRO_1MIN.txt.png"
Writing Gamera solar wind to bcwind.h5
Reading /glade/u/home/nikhilr/kaiju_engage/kaipy-private/kaipy/rcm/dktable
Reading /glade/u/home/nikhilr/kaiju_engage/kaipy-private/kaipy/rcm/wmutils/chorus_polynomial.txt
Dimension of parameters in Chorus wave model, Kp: 6 MLT: 97 L: 41 Ek: 155
Wrote RCM configuration to rcmconfig.h5
Template creation complete!
The PBS scripts ['./geospace-SPINUP.pbs', './geospace-WARMUP-01.pbs', './geospace-WARMUP-02.pbs', './geospace-01.pbs'] have been created, each with a corresponding XML file. To submit the jobs with the proper dependency (to ensure each segment runs in order), please run the script geospace_pbs.sh like this:
bash geospace_pbs.sh
``engage.py`` will then prompt you for the following additional information from ``tiegcmrun``:
.. code-block:: bash
Instructions:
-> Default Selected input parameter is given in GREEN
-> Warnings and Information are given in YELLOW
-> Errors are given in RED
-> Valid values (if any) are given in brackets eg. (value1 | value2 | value3)
-> Enter '?' for any input parameter to get a detailed description
Run Options:
User Mode = BASIC
Compile = False
Execute = False
Coupling = True
Engage = True
Directory of model [<YOUR_TIEGCMHOME_HERE>]:
Directory of Tiegcm Data Files [<YOUR_TIEGCMDATA_HERE>]:
Standalone Executable [<YOUR_TIEGCM_STANDALONE_EXECUTABLE_HERE>]:
Coupled Executable [<YOUR_TIEGCM_COUPLED_EXECUTABLE_HERE>]:
Low = 70, Medium = 140 , High = 200
F107 flux level for TIEGCM spin up (low|medium|high) [low]:
SOURCE file location [/glade/campaign/hao/itmodel/tiegcm3.0/new_data/source/junsol_f70.nc]:
If the SOURCE_START history is not found on the SOURCE file, the model will print an error message and stop.
Selected date in source file Example: (173,0,0,0) [173 0 0 0]:
STEP number [30]:
NSTEP_SUB number [10]:
Secondary Output Fields [['TN', 'UN', 'VN', 'NE', 'TEC', 'POTEN', 'Z', 'ZG']] / ENTER to go next:
High-latitude potential model that is going to be used (HEELIS|WEIMER) [HEELIS]:
If GPI_NCFILE is specified, then KP and POWER/CTPOTEN are skipped. If further POTENTIAL_MODEL is WEIMER and IMF_NCFILE is specified, then the Weimer model and aurora will be driven by the IMF data, and only F107 and F107A will be read from the GPI data file.
GPI file [/glade/campaign/hao/itmodel/tiegcm3.0/new_data/boundary_files/GPI/gpi_1960001-2024332.nc]:
After these inputs, the script interpolates source file for TIEGCM, and generates XML and
PBS files for the run, as well as a grid file for use in the model.
You should see output similar to this:
.. code-block:: bash
/glade/derecho/scratch/nikhilr/GTR58 exitsts
/glade/derecho/scratch/nikhilr/GTR58 exitsts
/glade/derecho/scratch/nikhilr/GTR58 exitsts
Interpolating primary file /glade/campaign/hao/itmodel/tiegcm3.0/new_data/source/junsol_f70.nc to create new primary file /glade/derecho/scratch/nikhilr/GTR58/tiegcm_standalone/geospace-tiegcm-standalone_prim.nc at horizontal resolution 2.5 and vertical resolution 0.25 with zitop 7.0.
Creating new primary file: /glade/derecho/scratch/nikhilr/GTR58/tiegcm_standalone/geospace-tiegcm-standalone_prim.nc
pbs_scripts = ['./geospace-01.pbs', './geospace-02.pbs']
submit_all_jobs_script = geospace_pbs.sh
You should now see the following files in your run directory:
.. code-block:: bash
ls
bcwind.h5 geospace-SPINUP.pbs lfmQ.h5
engage_parameters.json geospace-SPINUP.xml makeitso_parameters.json
geospace-01.inp geospace-WARMUP-01.pbs OMNI_HRO_1MIN.txt_bxFit.png
geospace-01.pbs geospace-WARMUP-01.xml OMNI_HRO_1MIN.txt.png
geospace-01.xml geospace-WARMUP-02.pbs rcmconfig.h5
geospace-02.inp geospace-WARMUP-02.xml tiegcm.exe
geospace-02.pbs geospace-WARMUP-03.pbs tiegcmrun_parameters.json
geospace-02.xml geospace-WARMUP-03.xml tiegcm_standalone
geospace.json geospace-WARMUP-04.pbs tiegcm.x
geospace_pbs.sh geospace-WARMUP-04.xml voltron_mpi.x
There are several types files created for each of the jobs, including:
* ``*.pbs``
These are the PBS scripts that will be submitted to the job scheduler to run
the segments of the simulation.
* ``*.xml``
These are the XML files that contain the parameters for GAMERA and RAIJU of the
segment.
* ``*.inp``
These are the namelist files that contain parameters for TIE-GCM of the segment.
* ``*.json``
These are the JSON files that contain the parameters for the simulation. They
are generated by the ``engage.py`` script with all the parameters required to run the
simulation.
The run is divided into segments:
* ``geospace-SPINUP.*``
This segment runs the GAMERA model to create the initial conditions for the
simulation. It is run first, and its output is used by the next segment.
* ``geospace-WARMUP-**.*``
These segments runs the GAMERA RAIJU model to "warm up" for for the coupled model execution.
The ``-01``, ``-02``, etc. suffixes indicate the segment number, and the
segments are run in order.
* ``tiegcm_standalone-**.*``
This segment runs the TIE-GCM model to create the initial conditions for the coupled model.
The ``-01`` to ``-08``. suffixes indicate the segment number, and the
segments are run in order.
* ``geospace-**.*``
These segments runs the GTR coupled modele. The ``-01``, ``-02``, etc.
suffixes indicate the segment number, and the segments are run
in order.
This image shows how the segments are run in order:
.. image:: GTRSegment.png
The image files are summaries of the CDAWeb data used in the initial condition
file (``bcwind.h5``). Those plots should look similar to this:
.. image:: Bx_fit.png
.. image:: sw.png
Finally, submit the model run using the script generated by ``engage.py``.
You will see the resulting PBS job ID (your job ID will differ from what is
shown below).
.. code-block:: bash
bash geospace_pbs.sh
9770226.desched1
9770227.desched1
9770228.desched1
9770229.desched1
9770230.desched1
9770231.desched1
9770232.desched1
9770233.desched1
9770234.desched1
9770235.desched1
9770236.desched1
9770237.desched1
9770238.desched1
9770239.desched1
9770240.desched1
Once the job is started in the queue, it should take about 80 minutes to run
(on ``derecho``). When complete, you will see many new HDF5 files in your
run directory, along with PBS housekeeping files and logs. The most important
files are (repeated upper-case letters in the names represent integer
strings):
* ``geospace_LLLLL_MMMMM_NNNNN_IIIII_JJJJJ_KKKKK.gam.h5``
These files contain the core MHD variables from the simulation, computed
by the GAMERA portion of the MAGE model. The strings ``LLLLL``, ``MMMMM``,
and ``NNNNN`` contain the number of subsections of the ``X``, ``Y``, and
``Z`` dimensions used to divide the domain among MPI ranks. The strings
``IIIII``, ``JJJJJ``, and ``KKKKK`` represent the MPI rank index along
each dimension.
* ``geospace.mix.h5``
This file contains the results from the
`REMIX <https://cgs.jhuapl.edu/Models/remix.php>`_ portion of the
`MAGE <https://cgs.jhuapl.edu/Models>`_ model.
* ``geospace.rcm.h5``
This file contains the results from the
`RCM <https://cgs.jhuapl.edu/Models/rcm.php>`_ portion of the
`MAGE <https://cgs.jhuapl.edu/Models>`_ model.
* ``geospace_sech_*.nc``
These are secondary output files that contain the results from the
`TIEGCM <https://tiegcm-docs.readthedocs.io/en/latest>`_ portion of the
`MAGE <https://cgs.jhuapl.edu/Models>`_ model.
* ``geospace_*.gam.Res.RRRRR.h5``
These are checkpoint files generated during the simulation which can be
used as restart points for future simulations.
* ``geospace_prim_*.nc``
These are the primary output files from the TIE-GCM portion of the model
that are designed as checkpoint files.
* ``geospace_temp_*.nc``
These are temporary output files from the TIE-GCM portion of the model
Visualizing the results
-----------------------
Now perform a quick visualization of the results from your model using the
``msphpic.py`` script, provided in the ``kaipy`` package.
.. code-block:: bash
msphpic.py -id geospace
This script will create a file called ``qkmsphpic.png``, which should look
similar to this:
.. image:: qkmsphpic.png

3
docs/source/running/index.rst
View File

@@ -39,5 +39,6 @@ Table of Contents
.. toctree::
:maxdepth: 1
geoQuickStart
geoGRQuickStart
geoGTRQuickStart
helioQuickStart

6
docs/source/tools/index.rst
View File

@@ -4,6 +4,12 @@ Analysis tools
These pages document the use of a set of compiled tools that come with the
``kaiju`` software.
Apart from the tools listed below:
- The ``kaipy`` Python package also provides a set of Python-based
tools for analyzing the output of the ``kaiju`` software.
These tools are documented in the `kaipy documentation <https://kaipy.readthedocs.io/en/latest/>`_.
- The ``gcmprocpy`` Python package provides a set of tools for processing
GCM output files. These tools are documented in the `gcmprocpy documentation <https://gcmprocpy.readthedocs.io/en/latest/>`_.
.. toctree::
:maxdepth: 1