Deploying MsPASS on an HPC cluster#

Overview#

First, by HPC (High-Performance Computing) we mean a cluster of multiple nodes linked by high speed interconnections designed for large-scale, parallel processing. If you are not familiar with modern concepts of this type of hardware and how they interact in HPC systems you should first do some background reading started with the section in our Getting Started pages found at MsPASS Virtual Cluster Concepts.

An axiom for working with MsPASS on a cluster is that any workflow you develop should first be prototyped on a desktop system. HPC systems are by definition designed to run large jobs that run on multiple nodes and use many cores. It is always best to test run any workflow on a subset of your data. For most people that is easiest on their office desktop machine. With that model you should plan to first construct the python code defining your workflow within a jupyter notebook on your desktop machine, transfer that notebook to the HPC system you want to use for the (presumably) much larger data set, and then face the new idioms of the HPC system you will be using.

Like your desktop system every HPC cluster has a set of local idioms. Examples, are file system directory names and variations in the software used to run jobs on the cluster. If there are other people in your institute who use MsPASS on the same cluster, your job will be much easier. In that case, you may be able to adapt an existing run script and/or configuration files from a colleague and bypass much of this document. If you are a pioneer in using MsPASS at your institution, you will need to read this more carefully.

At present there are two different approaches to running MsPASS on HPC cluster: (1) a python “launcher” method, and (2) a unix shell method. Unless you are an expert in the bash scripting language, most users will likely find the python launcher an easier way to get started. These two methods are described in separate sections below. First, however, it is necessary to describe some concepts you will need to understand to run a job with either method.

HPC MsPASS Concepts#

The block diagram in Figure Fig. 10 is an abstraction of the fundamental components required to run a MsPASS job on an HPC system. These components are not independent and it is necessary to understand how and when the different components in that diagram need to come into existence.

  1. HPC clusters use the idea of “batch processing” that dates to the earliest “supercompouters”, which were then called “mainframe computers”, of the 1960s. In modern HPC clusters that idea has evolved to job submission software. Users submit “jobs” to the cluster through a command line interface with the “job” being defined by a file that on all HPC systems today is a unix shell script. The “jobs” submitted are managed by a job scheduler (not to be confused with the dask/spark scheduler used in MsPASS) that manages which of many competing “jobs” will be run when. The submission process requires the user to specify the resources required for the job (e.g. number of nodes/cores and minimum memory requirements) and the amount of time those resources will be needed. Unlike time-shared access that is the norm for desktops, the job scheduler on an HPC cluster guarantees exclusive access to the requested resources for the time requested. Figure Fig. 10 illustrates the concept that the job scheduler selects a set of nodes, assigns them to your “job”, and starts to run your job script. In this case of MsPASS the actual physical hardware is abstracted to a set of services that define the framework. How those services are apportioned between the available nodes is a configuration issue. Because different jobs you may submit require vastly different resources (e.g. one node versus 10) this topic is more complicated for HPC systems than a desktop where the resources available are fixed.

  2. It is important to realize that you need to think of the MsPASS framework as four services: database (db), scheduler, worker(s), and frontend. The first thing your “job” needs to do to run a MsPASS workflow is to launch an instance of each of those services. MsPASS implements these services as a run argument to a standard “container”. On desktop systems the standard application to run a container is a package called docker. For a variety of reasons Docker is usually not permitted on multiuser HPC compute nodes. Instead the standard container launcher on HPC is apptainer. (Note until around 2023 this application was called singularity. Some sites may still run the legacy version.) There are currently two ways to launch these services: (a) a python launcher and (b) a unix shell script normally edited to become the job script.

  3. Once the services are all running a job normally start running a python script that defines the MsPASS workflow. That script can either be in the form of a plain python script or a sequence of run boxes in a jupyter notebook. In the first case the job has to run a python interpreter on the same node as the cluster scheduler. With a jupyter notebook the script is executed through the frontend service, which is normally launched on the same node as the scheduler.

../_images/HPCJobScheduler.png

Fig. 10 Conceptual diagram of how an HPC job scheduler allocates a set of nodes (3 in this example) as resources to construct a virtual cluster in MsPASS. The lines with arrows show illustrate that the physical hardware can be thought of as inputs used to define the virtual cluster illustrated here with the box with the label “MsPASS Abstraction”. How the inputs are allocated to build that cluster is the configuration problem that is the main topic of this section.#

Common Requirement#

At the time of this writing a software package called apptainer is the standard application for launching containerized applications like MsPASS on HPC clusters. Most multiuser centers do not permit the Docker daemon on compute nodes because of its privilege model. Apptainer, however, is compatible with Docker images in the sense that it can pull containers constructed with docker and build a file to run on the HPC cluster. That is the approach we use in this section.

Your first step is to log in to a login node (also called a head node), which is the host you use to access the cluster. Treat login nodes as a lightweight preparation and job-submission environment. Run MsPASS, container builds that consume substantial CPU or I/O, and interactive debugging only in a scheduled compute-node allocation, unless your center explicitly permits otherwise. HPC clusters commonly support a wide range of applications. As a result, all HPC clusters we know of have a software management system that controls what software is loaded in your environment. A typical incantation is the following:

module load apptainer

If apptainer is not available on your cluster, you will need to consult with system administrators to see if they run an alternative or are able to install apptainer. For smaller, specialized clusters you can consider building a local instance of MsPASS that doesn’t use the container if apptainer is a problem. That is not recommended for a variety of reasons but is possible.

Once your shell knows about Apptainer you will need to create an instance of the MsPASS container. Store the image on a shared filesystem intended for large project artifacts and visible from every compute node; $HOME and node-local temporary storage are usually poor choices. Customize the first line below for your site. Run the build on a workstation or in a scheduled compute allocation if it is too intensive for the center’s login-node policy.

export MSPASS_IMAGE_DIR=/path/to/shared/mspass/containers
mkdir -p "$MSPASS_IMAGE_DIR"
cd "$MSPASS_IMAGE_DIR"
apptainer build mspass_latest.sif docker://mspass/mspass

When the command exits you should now see the file mspass_latest.sif in $MSPASS_IMAGE_DIR. Note this is different from docker. Docker caches container data in its own work space. Apptainer creates a file, which in the case above we called “mspass_latest.sif”, containing the data required to launch the container.

Launching Services#

Python Launcher#

Installation#

The python launcher for MsPASS is distributed as part of a separate python package called mspass_launcher. If you are reading this you have probably already used this package to install mspass-desktop.

First check which Python interpreter you will use to run the launcher:

python --version

The published mspass-launcher 0.1.1 package declares Python 3.7 or newer; it does not require Python 3.10 or the match statement. Prefer a current Python version supported by your center. A typical module command is:

module load python/3.10.10

where the available version tag is site-specific. There are other options with virtual environments (notably pyenv and conda env), which can avoid conflicts with system packages.

Install mspass-launcher for that same interpreter as follows:

python -m pip install --user mspass-launcher

If you are using a virtual environment, activate it and omit --user.

Configuration#

Running the python launcher requires editing a configuration file. The normal expectation is that file has the magic name HPCClusterLauncher.yaml and an instance of that file is present in the job’s run directory. If a colleague at your institution has run MsPASS we advise you to use it as a starting point. Otherwise, copy the template installed with the same mspass-launcher version that will run the job. Release 0.1.1 and the current development branch use different configuration keys, so do not pair a PyPI installation with an unversioned main-branch template. The release 0.1.1 template is available for the published release.

Edit that file for your installation and the requirements of your workflow. Details on how to do that are found in this document.

Prepare a job script#

We assume the MsPASS script you will use to drive your data processing has been debugged previously and is in the form of either a jupyter notebook or a python run script. In either case it is important to realize the process must not require any user input since by definition an HPC job runs in “batch” mode.

A typical python job script would look like this:

#! /bin/bash

# ... Series of job scheduler resource definitions ...
module load apptainer
module load python/3.10.10  # optional - only needed if not using default python
cd *working_directory*
python << EOI
from mspass_launcher.hpc import HPCClusterLauncher
launcher = HPCClusterLauncher()
launcher.run("myscript.py")
EOI

where working_directory and the name “myscript.py” would by customized.

Note the HPCClusterLauncher.run method works with either a pure python script as in the above example or for a jupyter notebook file (“.ipynb” file name). We reiterate, however, that a notebook file must be runnable without intervention.

Finally, you should also be aware that the constructor for the HPCClusterLauncher object uses the default configuration file in the current directory. As you use MsPASS for a range of projects you will likely develop multiple configuration files that specify, for example, different numbers of workers. You can specify an alternative configuration file by giving a path to that file as arg0 or via the kwarg with key “configuration_file”. For example, pass the absolute path to an alternate file:

launcher = HPCClusterLauncher(
    configuration_file="/home/myusername/configurations/quartz_4_node.yaml"
)

Job submission#

Once you have a “job script” like the example above prepared you need to “submit” the job to the cluster. HPCClusterLauncher currently supports only Slurm, although other launch approaches may support other batch schedulers. The only difference is the incantations used to define resources and the command tool used to submit or monitor the progress of a job.

As an example, assume the “job script” we prepared has the file name “myproject.job”. If you log in to a login node on the cluster and cd to the directory where the job script is located you would submit it with “slurm” as follows:

sbatch myproject.job

noting that the “.job” in that file name is not required. It was done in this illustration to clarify it not a regular bash script but a Slurm job.

Understanding the python submission#

The way the HPCClusterLauncher object works is potentially confusing because an instance of the class has to manage the virtual cluster it launches and run a workflow on that cluster. Slurm submission happens before the launcher starts; the launcher itself does not submit the Slurm job. A wiring diagram may aid your understanding and is seen below in Figure Fig. 11.

../_images/HPCLauncherConcepts.png

Fig. 11 Simplified wiring diagram for MsPASS cluster abstraction using the python launcher for HPC systems (HPCClusterLauncher). The figure emphasizes that HPCClusterLauncher launches containers that define MsPASS services: db, scheduler, worker(s), and frontend. Note the “frontend” in this case is little more than an alias for python (illlustrated) because in this context when the HPCClusterLauncher.run method is called jupyter notebooks are first passed through nbconvert to convert the notebook to a python script before being run with an instance of python run on the frontend container.#

Important concepts that follow from the figure above are

  1. A typical job runs on multiple nodes. What service is run on what node is configurable. An exception is that the python run script always runs on what we call the primary node. It is normally the same node as the scheduler service.

  2. A corollary of 1 is that all serial processing will run on the primary node and compete for resources with any other service running on that node.

  3. The python interpreter running your job intercepts parallel constructs and submits them as task to “workers” managed by the “scheduler” service. The workers may be processes running on the same node as the scheduler or others designated as worker nodes.

  4. The job script that launches the processing runs on the primary node, but outside the container. That means the job script operates in a completely different python environment that the processing workflow that always run inside the containers. The is illustrated in the figure by showing the job script as running on the primary node but outside the boundaries of any of the service boxes. In words, the job script above launches an instance of a HPCClusterLauncher and then executes our workflow python/noebook file using the HPCClusterLauncher.run method.

Shell Script Run Option#

Overview#

Prior to the 2025 the only way to run MsPASS on any system was a totally command line driven method. That was true for both desktop and clusters. The mspass-desktop application described in Command Line Docker Desktop Operation and the HPCClusterLauncher class described above were developed to simplify running MsPASS. This section describes an alternative, now legacy approach some may prefer. It uses a lengthy shell script to launch MsPASS on an HPC cluster. The process is conceptually identical, but uses bash instead of python to handle the launch process. You may prefer this approach if you are an expert bash programmer or have used MsPASS previously in this mode.

Warning

The checked-in scripts/template and scripts/tacc_examples files are historical starting points, not turnkey job scripts. Their queues, module names, hostnames, tunnels, and process launchers must be adapted to the target system. For example, current TACC launching guidance requires ibrun rather than mpirun or mpiexec.

Get a Copy of Configuration Scripts#

You may first want to search the MsPASS configuration scripts If the system you are using has a folder there you should download the scripts from the appropriate folder and you should be able to proceed without having to dig too deep into this section. We assume here the file name convention is the same as that for the set in the folder template. If the file names for your institution are different you will have to do some additional work to puzzle out what was done. If there are deviations we recommend the author supply a README file.

If the files you need are not on github and you are aware of colleagues using mspass you may need to contact them and ask for their working startup scripts. If you are a trailblazer, then you will need to jump to the section below titled Setting Up Configuration Files on a new Cluster. You can then use the next section for reference when you are actively working with MsPASS on that system.

Build MsPASS Container with Apptainer#

You will need to build an Apptainer file. The process for this option is identical to that described above.

Edit template scripts#

Overview#

HPC clusters are designed to run large jobs and are not well-suited to interactive work. We reiterate that you are advised to develop your notebook on a desktop system using docker and a small subset of the data you need to process. This section assumes you have such a workflow debugged and ready to release on the full data set. It also assumes you aren’t a trailblazer and you have a template “job script” you or someone else at your institute has created that you only need to modify.

This section describes how to create a mspass “job” on HPC system as a set of unix shell scripts. We use that model because all HPC centers use the unix shell as the “job control” language. Old-timers from the days of mainframe computers from IBM and CDC may remember older, more primitive job control languages like the long dead IBM JCL (job control language). The concept is the same but today the language is a version of the unix shell. At present all our examples use the shell dialect called bash, but you are free to use any shell dialect supported by the cluster operating system.

Our standard template uses three shell scripts that work together to run a mspass workflow. The section heading titles below use the names of the template files. You can, of course change any of the file names provided you know how they are used.

mspass_setup.sh#

Before you run your first job you will almost certainly need to create a private copy of the template file mspass_setup.sh. This script does little more than define a set of shell environment variables to define where on the cluster file system the job can find your data and the mspass container. It also needs to define where a suite of work directories the different components of MsPASS need to utilize. The mspass_setup.sh file contains shell commands to set all the parameters that most users will need to customize. The idea is each user-dataset combination will normally require edits to this file. The rest of this section is arranged in the order of appearance of parameters in the template version of mspass_setup.sh show here:

#! /bin/bash

# See User's Manual for more guidance on setting these variables
export MSPASS_HOME=~/mspass
export MSPASS_CONTAINER=${MSPASS_HOME}/containers/mspass_latest.sif
# the container boots.  Usually an explicit path is best to avoid
export APPTAINER_BIND=/N/slate/pavlis,/N/scratch/pavlis

export MSPASS_WORK_DIR=/N/slate/pavlis/test_scripts
export MSPASS_DB_DIR=/N/scratch/pavlis/usarray/db
export MSPASS_LOG_DIR=/N/scratch/pavlis/usarray/logs
export MSPASS_WORKER_DIR=/N/scratch/pavlis/usarray/work


export HOSTNAME_BASE="carbonate.uits.iu.edu"

if [ -z $MSPASS_RUNSCRIPT ] ; then
  export MSPASS_RUNSCRIPT=/N/slate/pavlis/test_scripts/run_mspass.sh
fi

Notice that all this shell script does is set several environment variables that all begin with the string MSPASS_. The first one set is MSPASS_HOME. It is used like many software packages to define the home base for the software. In the MsPASS case it is used to define the location of the container needed to run MsPASS. If you created a private copy of the container in the section above you will not need to alter this parameter at all. If multiple people at your institute run MsPASS, there may be a master copy of the MsPASS container you can use in this definition. If so insert that path for this parameter.

The next line, which sets the environment variable APPTAINER_BIND, is a bit more obscure. Full understanding of why that incantation is necessary requires the concept of how to “bind” a file system to the container. A starting point is the Apptainer bind-path documentation. Briefly, the idea is much like a file system “mount” in unix. The comma separated list of directory names will be visible to your application as if it were a local file system. For the example above, that means your python script can open files in directories “/N/slate/pavlis” or “/N/scratch/pavlis”. Provided you have write permission to those directories you can also create file(s) and subdirectories under that mount point. You can also use source:destination bind syntax when the host and container paths should differ. Be cautious about binding over /home or another directory already populated in the image, because the bind hides the image’s original contents at that destination.

The four variables MSPASS_WORK_DIR, MSPASS_DB_DIR, MSPASS_LOG_DIR, and, MSPASS_WORKER_DIR define key directories needs to work. There use is as follows:

  • MSPASS_WORK_DIR is best viewed as the run directory. The run script will launch jupyter notebook with this directory as the top level directory. That means your notebook must be in this directory. It also serves as a top-level directory for defaults for MSPASS_DB_DIR, MSPASS_LOG_DIR, and, MSPASS_WORKER_DIR as noted in related items below.

  • MSPASS_DB_DIR is the work directory where MongoDB uses to store database data. If this variable is not set it defaults to $MSPASS_WORK_DIR/db.

  • MSPASS_LOG_DIR is used to write any log files. In MsPASS that means MongoDB and Dask/Spark. Any application that extends MsPASS may choose to log its own messages there. If so we recommend creating an appropriately named subdirectory under the one defined for MSPASS_LOG_DIR. If this variable is not set it defaults to $MSPASS_WORK_DIR/logs

  • MSPASS_WORKER_DIR is used by Dask/Spark as a scratch workspace. If this variable is not set it defaults to $MSPASS_WORK_DIR/work.

HOSTNAME_BASE should be set to the network subnet name the cluster runs in. That is usually necessary because all clusters we know of use a shortened name convention for individual nodes (i.e. the hostname has no “.” that is used for subnet naming.) If you are using an existing configuration file you almost certainly can use the value you inherited. Be warned that all HPC clusters we know use short names internally and the subnet definition is only needed if you plan to work interactively.

MSPASS_RUNSCRIPT defines what in section MsPASS Virtual Cluster Concepts is called a “virtual cluster”. The file is normally static for a particular cluster, although there may be mulitple options. e.g. the standard template file has versions with or without MongoDB “sharding”. For most users this file should be treated as static until performance becomes an issue and you find it necessary to do some advanced tuning. The last section of this document describes how that file may need to be modified if you are the first to use mspass on a cluster.

job_script.sh#

job_script.sh is the shell script you submit that runs your “job” on the cluster. Standard usage with Slurm as the workload manager to run the workflow in the Jupyter notebook file myworkflow.ipynb is:

sbatch job_script.sh myworkflow.ipynb

The template file assumes the file mspass_setup.sh defined above and the notebook file, myworkflow.ipynb, are present in the directory defined by MSPASS_WORK_DIR.

The only thing you would normally need to change in job_script.sh are the run parameters passed to Slurm with the #SBATCH lines at the top of the file. There are always cluster-dependent options you will need to understand before running a large job. Consult local documentation before setting these directives and submitting your first job.

Running a notebook interactively#

In some cases it may be necessary or helpful to develop your workflow, which in the MsPASS case means the code blocks in a Jupyter notebook, on the cluster. Even if you developed the notebook on a desktop it is often necessary to run the same test you prototyped on the HPC cluster before running a very large job. The simplest way to do that is to just run the notebook as above and verify you got the same answer you got on the version you debugged on your desktop. You may need to follow the procedure here if you need to do some additional interactive debugging or your desktop has limitations (e.g. memory size) that you cannot simulate on your desktop. This section describes the basic concepts required to do that. Details depend on how a browser on your local computer reaches the Jupyter server. The complexity is a reason to prefer a supported portal when one is available.

Prefer a center-supported portal or a scheduler-backed interactive allocation. For example, current TACC systems provide Jupyter through the TACC Analysis Portal. Never start Jupyter, MsPASS services, or a container workflow directly on a shared login node.

The procedure for running MsPASS interactively is similar to that for running containers on a desktop system found in Command Line Docker Desktop Operation. There are two key differences: (1) you launch MsPASS with apptainer (or something else) instead of docker and (2) there are a lot of potential network issues this manual cannot fully cover. This subsection is mainly aimed to address the first. We provide only some initial suggestions below for potential networking issues.

We assume that the interactive job you need to run is suitable for the all-in-one configuration we use in Docker. In that configuration all the individual MsPASS components are run as different processes in one container on one node. Our template script for setup is called single_node.sh. A method to launch MsPASS in that mode with Slurm would be to enter the following command:

sbatch single_node.sh

You should then use the squeue Slurm command to monitor when your job starts or watch for the appearance of the output file defined by slurm commands in single_node.sh. Typically use the unix cat command to print the output file. The output is similar to what one sees with docker run. The following is retained as historical example output generated in 2023 on the Indiana University cluster called “carbonate”; its Singularity version, module list, Jupyter version, hostnames, and URLs are not current instructions:

singularity version 3.6.4 loaded.
Currently Loaded Modulefiles:
1) quota/1.8                      8) boost/gnu/1.72.0
2) git/2.13.0                     9) gcc/9.1.0
3) xalt/2.10.30                  10) openblas/0.3.3
4) core                          11) intel/19.0.5
5) hpss/8.3_u4                   12) totalview/2020.0.25
6) gsl/gnu/2.6                   13) singularity/3.6.4
7) cmake/gnu/3.18.4              14) openmpi/intel/4.0.1(default)
/N/slate/pavlis/usarray
Thu Jan 26 10:43:56 EST 2023
{"t":{"$date":"2023-01-26T15:44:10.476Z"},"s":"I",  "c":"CONTROL",  "id":20697,   "ctx":"main","msg":"Renamed existing log file","attr":{"oldLogPath":"/N/slate/pavlis/usarray/logs/mongo_log","newLogPath":"/N/slate/pavlis/usarray/logs/mongo_log.2023-01-26T15-44-10"}}
[I 10:44:16.973 NotebookApp] Serving notebooks from local directory: /N/slate/pavlis/usarray
[I 10:44:16.974 NotebookApp] Jupyter Notebook 6.2.0 is running at:
[I 10:44:16.974 NotebookApp] http://c4:8888/?token=e7464f3b156b27efcaf2c9e52197b40068c5eefd8231a955
[I 10:44:16.974 NotebookApp]  or http://127.0.0.1:8888/?token=e7464f3b156b27efcaf2c9e52197b40068c5eefd8231a955
[I 10:44:16.974 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[C 10:44:17.036 NotebookApp]

To access the notebook, open this file in a browser:
    file:///N/slate/pavlis/usarray/.local/share/jupyter/runtime/nbserver-11604-open.html
Or copy and paste one of these URLs:
    http://c4:8888/?token=e7464f3b156b27efcaf2c9e52197b40068c5eefd8231a955
 or http://127.0.0.1:8888/?token=e7464f3b156b27efcaf2c9e52197b40068c5eefd8231a955

Like the docker case the information to connect to Jupyter is found in the last few lines. For the above example the key line is:

[I 10:44:16.974 NotebookApp] http://c4:8888/?token=e7464f3b156b27efcaf2c9e52197b40068c5eefd8231a955

In this case c4 was the shortened hostname used inside that cluster. At the time of this historical example, the site’s network mapping used the following URL:

http://c4.uits.iu.edu:8888/?token=e7464f3b156b27efcaf2c9e52197b40068c5eefd8231a955

Do not infer a current connection recipe from that URL. Use the site’s portal or documented forwarding procedure, keep the Jupyter token secret, and do not expose Jupyter or the Dask dashboard on a shared interface. Connection details are site-dependent; see the networking subsection below.

There is a final, very important warning when running a “job” interactively started with slurm. When you finish the interactive work you should kill your running “job” immediately. If you don’t the node will sit around doing nothing until the time limit you specified expires. If you ignore this warning you can quickly burn your entire allocation with no results. With Slurm the way to terminate a job that you own is:

squeue -u myusername
scancel jobid

Run that pair of commands sequentially, substituting your user name in the first. The output will show an ID with a format something like this:

JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
3298684   general   mspass   pavlis  R       4:33      1 c4

For this example jobid is 3298684. That job is “killed” by the command scancel 3298684.

Finally, some clusters provide a simplified procedure to run interactive jobs through a portal or science gateway. A historical Indiana University example used a “Research Desktop” (RED) system that provided a remote Linux desktop. In that case, running an interactive job is similar to running a container on a desktop, except the portal must first allocate one or more compute nodes. After the portal or scheduler has allocated a compute node, you can run the configuration shell script interactively. Do not source it in a terminal that is still attached to a shared login or gateway node. Inside the allocated compute-node terminal you can then run:

source single_node.sh

which should generate an output similar to that above for the sbatch example. Connect to Jupyter only through the mechanism documented by that gateway.

If running on distributed nodes, a manually constructed database client should use the database host defined by MSPASS_DB_ADDRESS (not the scheduler address). For example:

from mspasspy.db.client import DBClient
import os
dbclient=DBClient(os.environ.get("MSPASS_DB_ADDRESS"))

The standard launch scripts set both service addresses for the frontend; they may happen to name the same primary node, but they have different meanings.

Setting Up Configuration Files on a new Cluster#
Overview#

If you are a trailblazer at your institution and need to configure MsPASS for your local cluster, you may want to first review the material in this User’s Manual found in the section MsPASS Virtual Cluster Concepts. That provides some fundamental concepts on HPC systems and how those concepts are abstracted in MsPASS to produce a virtual cluster. This section focuses on the nuts and bolts of what you might have to change in your local configuration. The descriptions here are limited to the simpler situation with a single instance of the database server (not “sharded”). Sharding is an advanced topic and we assume if you are needing that feature you are hardy enough you solve the problem yourself. This section assumes you have a copy of the file in the mspass scripts/template directory called “run_mspass.sh”. You may also find it useful to compare that file to the examples for specific sites.

The “Role” Concept#

In the section titled MsPASS Virtual Cluster Concepts we discuss in detail the abstraction we used in MsPASS to define what we call a “virtual cluster”. A key idea in that abstraction is a set of functional boxes illustrated in Fig. 12. The function each box illustrated there is defined by what we call its “role”. The keywords defining “role”, with one line descriptions of what functionality they enable are the followings:

  • db creates and manages the MongoDB server

  • scheduler is the dask or spark manager that controls data flow to and from workers

  • worker task that do all the computational task.

  • frontend runs Jupyter for an interactive session or the master Python script for a batch workflow.

Note the configuration illustrated in Fig. 12 is a graphical illustration of that created with the template run_mspass.sh script.

../_images/FiveNodeExampleComposite.jpg

Fig. 12 Block diagram of virtual cluster that is defined by the run_mspass.sh template file. This illustrates the geometry for five nodes, but the configuration is open-ended. If more than 5 nodes are used any additional nodes will be set with role == “worker”. Notice with this configuration all roles other than worker are run on the same node as the job script is executed illustrated here as “node 1”#

How Different Roles are Run#

Notice from Fig. 12 that all 4 roles are launched as separate instances of the container. In the script they are all launched with variations of this following:

SING_COM="apptainer run $MSPASS_CONTAINER"
APPTAINERENV_MSPASS_WORK_DIR=$WORK_DIR \
     APPTAINERENV_MSPASS_ROLE=scheduler $SING_COM &

where we illustrate the definition of the symbol SING_COM for clarity only. In the actual script that line appears earlier. The above is the actual launch line for the scheduler. Note the following that are used when each instance of the container is launched:

  • The run command is preceded by a set of shell variable definitions that all begin with the keyword APPTAINERENV. An odd feature of apptainer is any shell symbol it detects that begin with APPTAINERENV_ have that prefix stripped and the result posted to a shell environment variable that is available to the container boot script, which in MsPASS is called start-mspass.sh, (That shell script is not something you as user would ever change but it may be instructive to look at that file to understand this setup. That file can be found in the mspass github site at the top of the directory chain.) For example, when the above line is executed the variable MSPASS_ROLE is set to “scheduler”.

  • Notice the container is launched as a background process using the standard unix shell “&” idiom. Notice that all lines that execute $SING_COM contain the “&” symbol EXCEPT the jupyter notebook server that is the last line in the script. That syntax is important. It cause the shell running the script to block until the notebook exits. When the master job script exits apptainer does the housecleaning to kill all the running containers on multiple nodes running in the background.

  • The instances of the container for the db and frontend role launch are similar to the scheduler example above but with different APPTAINERENV inputs. The worker launching is different, however, and is the topic of the next section.

Launching Workers#

Launching workers is linked to a fundamental problem you will face in adapting the template script to a different cluster: node-to-node communications. There are two low-level issues you will need to understand before proceeding:

  1. How are nodes addressed? i.e. what symbolic name does node A need to know to talk to node B?

  2. What communication channel should be used between nodes?

For the first, all the examples we know use a short form of hostname addressing that strips a subnet description. You are probably familiar with this idea working on any local network. e.g. the machine in my office has the long name “quakes.geology.indiana.edu”. That name resolves as a valid hostname on the internet because it is advertised by campus name servers. Within my department’s subnet, however, I can reference to the same machine with the simpler name “quakes”. The same shorthand is standard on any clusters we know of so short hostnames are the norm.

That background is necessary to explain this incantation you will find in run_mspass.sh:

NODE_HOSTNAME=`hostname -s`
WORKER_LIST=`scontrol show hostname ${SLURM_NODELIST} | \
           awk -vORS=, -v hostvar="$NODE_HOSTNAME" '{ if ($0!=hostvar) print $0 }' | \
           sed 's/,$/\n/'`

The first line returns the human readable name of the node on which the script is being executed. The -s, which is mnemonic for short, strips the subnet name from the fully qualified hostname. As noted above it may not be required on your site as it is common to use only the base name to reference nodes.

The second line, which truly deserved the incantation title, sets the shell variable WORKER_LIST to a comma-delimited list of the hostname of all nodes allocated to this job excluding the node running the script (result of the hostname command). To help clarify here is the section of output produced by this script run with four nodes on an Indiana University cluster:

Lauching scheduler on primary node
c23,c31,c41

where c23, c31, and c41 are the hostnames of the three compute nodes slurm assigned to this job. That list is used to launch each worker in another shell incantation that follows immediately after the above:

APPTAINERENV_MSPASS_WORK_DIR=$WORK_DIR \
  APPTAINERENV_MSPASS_SCHEDULER_ADDRESS=$NODE_HOSTNAME \
  APPTAINERENV_MSPASS_ROLE=worker \
  mpiexec -n $((SLURM_NNODES-1)) -host $WORKER_LIST $SING_COM &

This historical template uses the Open MPI command-line tool mpiexec to launch the container on all the nodes except the first one in the list. MPI is used only as a convenient way to launch background processes on nodes Slurm assigned to the job. This syntax is site- and MPI-specific: current TACC systems require ibrun and explicitly advise against mpirun or mpiexec. Other centers may require srun or another approved launcher. Do not replace it with ad hoc SSH commands, and never launch work on nodes outside the active allocation. Expect to modify and test this line for the target system.

Communications#

Last, but far from least you may need to sort out some fundamental issues about how networking is implemented on your cluster. There are two different issues you may need to consider:

  1. Are there any network communication restrictions between compute nodes? Dask and Spark have different communication setups described in the links in this sentence. The general pattern seems to be that clusters are normally configured to have completely open communication between nodes within the cluster but are appropriately paranoid about connections with the outside world. i.e. you probably won’t need to worry about connectivity of the compute nodes, but problems are not inconceivable.

  2. A problem you are guaranteed to face is how to connect to a job running on the cluster. The simplest example is needing to connect to the jupyter notebook server for an interactive run. We reiterate that isn’t a great idea, but you will likely eventually need to use that feature to solve some problem that you can’t solve easily with batch submissions. A more universal need is to run real-time dask diagnostics. These are an important tool to understand bottlenecks in a parallel workflow that are limiting performance. For dask diagnostics to work you will need to connect on some port (default is 8787) to the node running the scheduler. The fundamental problem both connections face is that clusters are normally accessible from outside only through “login nodes” (also sometimes called head nodes). The login nodes are sometimes called a network “gateway” to the cluster, which should not be confused with the something more properly called a “scientific gateway”. The later is a simplified access method to reduce the very kind of complexity discussed in this section for normal humans.

The historical template addressed item 2 with reverse SSH tunnels. The following block is retained to explain that design, not as a current recipe.

Warning

Do not copy this block without approval and current instructions from your computing center. It assumes obsolete TACC hostnames and reverse forwards that may expose services on a shared host. Current TACC users should prefer the TACC Analysis Portal.

NODE_HOSTNAME=`hostname -s`
LOGIN_PORT=`echo $NODE_HOSTNAME | perl -ne 'print (($2+1).$3.$1) if /c\d(\d\d)-(\d)(\d\d)/;'`
STATUS_PORT=`echo "$LOGIN_PORT + 1" | bc -l`
echo "got login node port $LOGIN_PORT"

NUMBER_LOGIN_NODES=4
LOGIN_NODE_BASENAME=login
for i in `seq $NUMBER_LOGIN_NODES`; do
  ssh -q -f -g -N -R $LOGIN_PORT:$NODE_HOSTNAME:8888 $LOGIN_NODE_BASENAME$i
  ssh -q -f -g -N -R $STATUS_PORT:$NODE_HOSTNAME:8787 $LOGIN_NODE_BASENAME$i
done

The first section using Perl attempted to avoid a port collision on a shared login node. The historical TACC example converted the compute node’s hostname to a port number because the allocated compute-node name was unique to the job. This is not a portable way to reserve a port. Do not select fixed ports on a shared host or expose a reverse forward unless the site documents that procedure.

The second section handled a historical environment with four login nodes named login1 through login4. The SSH lines set up reverse tunnels from the compute node to those login nodes. Modern centers commonly route login connections with DNS or a load balancer; Slurm does not choose the login node. If a center still supports this design, follow its current instructions, explicitly bind the remote listening address to loopback, and omit the irrelevant -g option. In the old mapping, Jupyter port 8888 was forwarded to LOGIN_PORT and Dask dashboard port 8787 to STATUS_PORT.

We emphasize that none of the network complexity is required in two situations we know of:

  1. If you only intend to run batch jobs, then connections to the outside will not be needed and you can delete all the network stuff from the template. In fact, we recommend you prepare a separate run script, which you might call run_mspass_batch.sh that simply deletes all the network stuff above.

  2. Some sites may have a science gateway or portal that provides a mechanism to run jobs interactively on the cluster. The example noted earlier used at Indiana called “RED” is an example. With RED you launch a window on your desktop that behaves as if you were at the system console for the login node. In that situation user-created SSH tunnels may not be necessary. Follow current gateway documentation rather than assuming that ports 8888 and 8787 are directly reachable.