With the current cluster configuration, you normally do not need to specify a queue or partition when submitting new compute jobs, as this will be done automatically by Slurm, depending on the job's properties (e.g. the run time or special resources like accelerators).

Members of multiple projects however, should make sure to choose the proper project when submitting new jobs (e.g. with the parameter -A <project name>).

• sbatch <job script>
  This puts a new job in the queue. Please refer sbatch-parameters for further important parameters of commands and job scripts.
  More detailed examples of job scripts are available under “script”.
• squeue
  This shows an overview of all your active and waiting jobs of the job queue.
• csqueue
  This shows a global overview of all the active and waiting jobs of job queue.
• sjobs <Job-ID>
  This is a special TU Darmstadt script for showing detailed information about all your pending and running jobs or the job with the given ID.
• scancel <Job-ID>
  This deletes a job from the queue or terminates an active job.
  • scancel -u $USER
    Deletes/terminates all your own jobs.
• csreport
  This is a special TU Darmstadt script for showing the resource usage for each of your projects in the last months. All values are shown in core*hours per month, and are thus percent-related to your monthly share (not to your project's total budget).
• For the current month you also see your own part of usage for each of your projects (important for projects with multiple users).
  
  • sreport
    This is the standard Slurm command and shows the resource usage separated for each of your projects.
    ^{Attention: without “-t hours”, the values are given in core*minutes.}
    • In addition you can get a report of a specific month or any time period. For that you need to give the parameters “cluster account” (verbatim) and the start and end date.
      The following example shows how to get a core*h report of April 2024:
      sreport cluster account Start=2024-04-01 End=2024-05-01 -t hours
• csum
  This is a special TU Darmstadt script for showing the resource usage in total for each of your projects (in comparison to the approved value). This command shows all values in core*hours.

To enhance reproducibility, we recommend providing all parameters inside the job script as “#SBATCH …” pragma (instead of using sbatch command line parameters). This way, even rather old job scripts clearly document the conditions the job ran under.

Examples of different job scripts (MPI, OpenMP, MPI+OpenMP=hybrid, GPU) can be found down below.

Here, only the most important pragmas are given. You can find a complete list of parameters using the command ''man sbatch'' on the login nodes .

-A project_name
With this option, you choose the project the core hours used will be accounted on.
Attention: If you omit this pragma, the core hours used will be accounted on your default project (typically your first or main project), which may or may not be intended!

-J job_name
This gives the job a more descriptive name.
Referable as %x (in #SBATCH … pragmas) and as $SLURM_JOB_NAME (in the job script's payload).

--mail-type=BEGIN
Send an email at begin of job.
--mail-type=END
Send an email at end or termination of job.
--mail-type=ALL
Send an email at both events (and in some other special cases).

Please note: if you submit a lot of distinct jobs separately, at least the same number of emails will be generated. In the past, this problem has caused the mail servers of the TU Darmstadt to be blacklisted as “spamming hosts” by several mail and internet service providers, refusing to receive any further mail from the TU Darmstadt.
The mail and groupware team of the HRZ was having a lot of efforts to revert this.

Avoid this by

• using job arrays (#SBATCH -a 1-100 for 100 similar jobs)
• --mail-type=NONE – instead, use “squeue” to see all your jobs still running or finished.

Please also avoid setting “--mail-user=<yourMailAddress>”, as our system determines your mail address automatically based on your TU-ID, and typos in this parameter cause unnecessary workload.

-o /path/to/outfile_name
This writes the standard output (STDOUT) of the whole job script in the designated file.
-e /path/to/errfile_name
This writes the error channel (STDERR) of the whole job script in the designated file.

For both options, we recommend to use the full pathname and/or to use %. file name variables, to avoid overwriting other job's files.

--gres=class:type:# accelerator specification, eg. GPUs
(if not specified, the defaults are: type=any and #=1)

• --gres=gpu
  ^{requests 1 of any GPU accelerator cards (not recommended--can be either Nvidia or PVC or AMD)}
• --gres=gpu:v100
  ^{requests 1 NVidia “Volta 100” card}
• --gres=gpu:a100:3
  ^{requests 3 NVidia “Ampere 100” cards}
• --gres=gpu:pvc128g:2
  ^{requests 2 Intel “Ponte Vecchio” GPUs with 128 GByte G-RAM (PVC usage info (opens in new tab) )}
• --gres=gpu:mi300x:6
  ^{requests 6 AMD MI300X GPUs with 192 GByte G-RAM each (MI300X usage info (opens in new tab) )}

^{To have your job scripts (and programs) adapt automatically to the amount of (requested) GPUs, you can use the variable $SLURM_GPUS_ON_NODE wherever your programs expect the number of GPUs to use, ie.
“myCUDAprogram --num-devices=$SLURM_GPUS_ON_NODE”.}

Since you always get whole GPU cards allocated, there is no way of asking for a certain amount of tensor cores or shaders or G-RAM (GPU memory) in a GPU. When allocated, you are free to use each and every resource inside that allocated GPU card at will.

If you need more than one GPU compute node for distributed Machine/Deep Learning (eg. using “horovod”), the job needs to request several GPU nodes explicitly using -N # (with # = 2-8). Consequently, the number of tasks requested with -n # needs to be equal or higher than the number of nodes.

Since “GRes” are per node, you should not exceed --gres=gpu:4 (except when targeting the DGX with 8 GPUs or the AMD MI300X GPUs), even when using several 4-GPU-nodes.

In principle. it is possible to work interactively on compute nodes, though it is not advised for regular work: due to many pending jobs and a mostly fully used-up cluster, resources will not be available immediately.

If interactive work on compute nodes in fact is necessary, it can be requested with the srun command and the --pty /bin/bash option. The three mandatory parameters are -t (time), -n (No. of tasks) and --mem-per-cpu= (memory per task), and need to be supplied to the srun command, too. Optional parameters like features and mail options can also be given on the command line.

Example

srun -t15 -n4 --mem-per-cpu=500 --pty /bin/bash

In general, a direct login to the compute nodes is not possible.

However, during execution of your own job(s), you are entitled to login to the executing compute nodes (from a login node, not from the internet).

That is to run “top”, “htop” or “strace” or similar utilities, and in general to see the behaviour of your job(s) at first-hand.

^{This works only from the login node, as the compute nodes are not directly connected to the internet.}

To get the (list of) compute node(s) executing your job, run

squeue -t RUNNING

Then hop onto one of them (from the login node only) by either executing

ssh <name of compute node as shown in NODELIST>

In case you requested a total of one compute node, you can also achieve the same with

srun --jobid=<yourJobID> --pty bash

without prior extraction of the NODELIST.

In case of multi-node MPI jobs, this command would put you on any one of your compute nodes.

In order to go to other nodes involved, the “NODELIST” from the squeue output needs to be decomposed to get distinct host names. For example, the node list “mpsc0[301,307,412-413]” expands to

mpsc0301
mpsc0307
mpsc0412
mpsc0413

which you are all entitled to log in to with “ssh mpsc0…” while these are still executing your job.

To have a detailed look on how (and if) your job is using the assigned resources, you can use commands like “top” or “htop” or your own favorite linux tools.

If your job ends while you are still logged into one of its compute nodes, you will be logged out automatically, ending up back on the login node.