05_batch_jobs.md

---

theme: csc-eurocc-2019
lang: en

---

The batch job system in CSC's HPC environment {.title}


All materials (c) 2020-2025 by CSC – IT Center for Science Ltd. This work is licensed under a **Creative Commons Attribution-ShareAlike** 4.0 Unported License, [http://creativecommons.org/licenses/by-sa/4.0/](http://creativecommons.org/licenses/by-sa/4.0/)

What is a batch job? 1/2

• On a laptop you might be used to start a program (job) by clicking an icon, which starts the job instantly

• If we start many jobs at the same time, we occasionally encounter problems like running out of memory etc.

• In an HPC environment, the computer is shared among hundreds to thousands of other users who all have different resource needs

• HPC batch jobs include a resource request, which corresponds to an estimate of how much resources the job is expected to use

What is a batch job? 2/2

• A batch job consists of two parts: A resource request and the actual computing step

• A job does not start directly, but is sent to a queue

• Depending on the requested resources and current load on the system, the job may need to wait for a while before starting

• At CSC (and HPC systems in general), all heavy computing must be done via batch jobs (see our usage policy)

What is a batch job system?

• A resource management system that keeps track of all jobs that use, or would like to use, the computing resources

• Aims to share the resources in an efficient and fair way among all users

• Optimizes resource usage by filling the compute nodes so that there will be as little idling resources as possible

Queueing and fair share of resources

• A job is queued and starts when the requested resources become available

• The order in which the queued jobs start depends on their priority and currently available resources

• At CSC, the priority is configured to use "fair share"

  • The initial priority of a job decreases if the user has recently run lots of jobs

  • Over time (while queueing) its priority increases and eventually it will run

  • Some queues have a lower priority (e.g. longrun -- use shorter if you can!)

• See our documentation for more information on Getting started with running batch jobs on Puhti/Mahti and LUMI.

Schema of how the batch job scheduler works

The batch job system in CSC's HPC environment

• CSC uses a batch job system called Slurm to manage resources

• Slurm is used to control how the overall computing resources are shared among all jobs in an efficient and fair manner

• Slurm controls how a single job request is allocated resources, such as:

  • computing time

  • number of cores

  • amount of memory

  • other resources like GPUs, local disk, etc.

• Getting started with Slurm batch jobs on Puhti/Mahti and LUMI

An example serial batch job script for Puhti

• A batch script is a shell script (bash) that consists of two sections:

  • Resource requests flagged with #SBATCH and the actual computing step(s)

#!/bin/bash
#SBATCH --time=00:01:00             # Defines the max time the job can run
#SBATCH --partition=test            # Defines the queue in which to run the job
#SBATCH --ntasks=1                  # Defines the number of tasks (processes)
#SBATCH --cpus-per-task=1           # Total number of cores is ntasks * cpus-per-task
#SBATCH --account=<project>         # Defines the billing project, e.g. project_2001234 (mandatory field)

srun echo "Hello $USER! You are on node $HOSTNAME"

• The options are described in Docs CSC: Create Puhti batch jobs

  • The actual program is launched using the srun command

  • The content above could be copied into a file serial.bash and submitted to the queue with sbatch simple_serial.bash

Using an application-specific batch script template

• The application pages in Docs CSC contain example scripts for some software

• Use these as the starting point for your own scripts

• They have been tested and optimized (although for minimal resources) for that application

  • Consult the official manual or other examples to adapt to your own needs

  • Ask for support: [email protected]

{width=100%}

Submitting, cancelling and status of batch jobs

• A batch job script is submitted to the queue with the command:

  • sbatch example_job.sh

• List all your jobs that are queuing/running:

  • squeue -u $USER

• Detailed info of a queuing/running job:

  • scontrol show job <jobid>

• A job can be deleted using the command:

  • scancel <jobid>

• Display the resource usage and efficiency of a completed job:

  • seff <jobid>

Available batch job partitions

• The available batch job partitions are listed in Docs CSC

  • The Slurm partitions available on LUMI are detailed in the LUMI documentation.

• In order to use the resources efficiently, it is important to estimate the resource request as accurately as possible

• By avoiding excessive "just-in-case" requests, the job will start earlier

Different types of HPC jobs

• Typically, an HPC job can be classified as serial, parallel or GPU, depending on the main requested resources

• The following slides will present an overview of different job types

• A serial job is the simplest type of job, whereas parallel and GPU jobs require advanced software and programming methods to fully utilise their capacity

  • Note that GPU-jobs are in principle also parallel, but they run on different hardware (GPUs instead of CPUs) and are programmed differently

• If you use pre-installed applications, please ensure what kind of resources they need to run efficiently (serial, parallel or GPU)

HPC serial jobs

• A serial software can only use a single core, so don't reserve more!

• Why could your serial job benefit from being run using CSC's resources instead of on your own computer?

  • Part of a larger workflow (high-throughput computing)

  • Avoid data transfer between the supercomputers and your own computer

  • Data sharing among other project members

  • CSC's software licensing

  • Pre-installed software

  • Memory and/or disk demands

Running multiple serial jobs

• You can utilize HPC resources for running multiple independent serial jobs at the same time (task farming)

  • Array jobs

  • Other high-throughput tools

• On Mahti, pure serial resources are only available in small partition

  • Some tools, e.g. HyperQueue, can make a set of serial jobs suitable also for medium partition

  • But, the workflow needs to fill (at least) one Mahti node (128 cores) and keep the CPUs busy for the job duration

• When running many jobs, make sure that you don't overload the batch queue system or the parallel file system (mind your I/O and job steps)!

HPC parallel jobs

• A parallel job distributes the calculation over several cores in order to achieve a shorter wall-time (and/or a larger allocatable memory)

  • The total computational problem is divided into subtasks, which are processed by each core in parallel

• There are two major parallelization standards: OpenMP and MPI

  • Note, depending on the parallellization scheme there is a slight difference between how the resource reservation is done

• Batch job scripts for Puhti (how to create and examples), Mahti (how to create and examples) and LUMI (quickstart, CPU and GPU examples)

• The best starting point: Software specific batch scripts in Docs CSC

HPC GPU jobs

• A graphics processing unit (GPU, a graphics card), is capable of doing a certain type of simultaneous calculations very efficiently

• In order to take advantage of this power, an application must be (re)programmed to adapt to how the GPUs process data

• CSC's GPU resources on Puhti and Mahti are relatively scarce and should be used only by applications that really benefit from GPUs

  • A GPU on Puhti/Mahti uses 60 times more billing units than a single CPU core - see above for performance requirements

  • In practice, 1-10 CPUs (but not more) should be allocated per GPU on Puhti

  • Note that LUMI-G has a massive GPU capacity available, which is also "cheaper" as measured in BUs compared to Puhti/Mahti

Interactive jobs

• When you login to CSC's supercomputers, you end up on one of the login nodes of the supercomputer

  • These login nodes are shared by all users and they are not intended for heavy computing

• If you have a heavier job that still requires interaction

  • Request resources from the interactive partition using the sinteractive command

  • This will open an interactive shell where you can perform your computations directly on a compute node instead of the login node

• Interactive jobs and GUIs can also be launched in the web interfaces

  • Jupyter notebooks, RStudio, TensorBoard, MATLAB, VSCode, ...