FAQs – Hochleistungsrechnen

FAQ

Frequently Asked Questions – Access

User Account

That can be caused by two basic reasons: the login node is not answering at all (perhaps due to being down or having network problems), or it answers but denies you access.

Try another login node (the one you tried may be down)
Have a look at our news page for possible maintenance downtimes or unexpected outages.
Check your professional email inbox (and spam folder) for mails with the subject “Your account on the Lichtenberg High Performance Computer”, warning of or indicating your recent account's expiry.
In case of “Permission denied” or “Incorrect password” (the login node answers, but denies you access), even from different login nodes:
- check the language settings of your keyboard for inadvertent changes. English keyboard layout has Y and Z inverted, and does not sport german umlaut's.
- make sure to log in with your TU-ID. Some ssh programs guess your login name from what you are on the local system, ie. the windows ssh client likes to transmit “ADS-DOMAIN\yourWindowsName” and that can't work. Either use
  ssh <TU-ID>@lclusterXX.hrz.tu-darmstadt.de or
  ssh -l <TU-ID> lclusterXX.hrz.tu-darmstadt.de
- try changing your TUID password, as this is sync'ed to the HLR a few seconds later.
Read the (ssh) error message in its entirety. Sometimes it even explains how to fix the actual problem.
Try to log in explicitly with IPv4 or IPv6:
ssh -XC –4 <TU-ID@>lcluster15.hrz.tu-darmstadt.de
ssh -XC –6 <TU-ID@>lcluster17.hrz.tu-darmstadt.de
Already running the most recent version of your ssh/scp access program (PuTTY / KiTTY / BitVise / WinSCP / FileZilla)?
From time to time, we refine the allowed (ie. considered safe) list of ssh key ciphers and negotiation algorithms, and some older program (versions) might be unable to cope with them.
Have you recently made any changes in your login startup scripts .bashrc or .bash_profile? When failing, commands in these startup scripts can cause the bash to end prematurely, and that might look like you are denied access.
If only interactive logins are affected, you can try accessing the likely culprit with your scp program (BitVise / WinSCP / FileZilla), and either
- download it, comment the recent changes and re-upload it
  or
- delete the troublemaker (and then restoring your last version from .snapshots/)

If nothing of the above works out: open a ticket, always mentioning your TUID (no enrollment/matricle no) and preferably from your professional TUDa email address.

Avoid sending “screenshots” of your terminal window as pictures (jpg/png or the like). A simple copy&paste in text form of your login attempts and the resulting error message is sufficient.

If there isn't any explanatory error message, please use the “verbose” mode by running “ssh -vv …” and append the output to the ticket mail (again please not as screenshot/picture, but as text).

Projects

Using the "member“ command (new since Nov 2022) without parameters, you see a list of all your memberships in HPC projects.

Additionally, your currently active projects / memberships are recorded nightly in a file called .project in your HOME directory:

 cat $HOME/.project

(see also ”Hints " about project application).

Frequently Asked Questions – batch scheduling system

Preparing Jobs

Yes, that is possible with the parameter “--test-only” for “sbatch”:

sbatch --test-only submit_tests
[E] sbatch: Job 12…78 to start at yyyy-mm-ddThh:mm:ss using XX processors on nodes mpsc[000y-000z] in partition PPPP

If you get back this kind of message, Slurm would accept this job script, and would run the job most likely at the given time (estimated!).

While apparently classified as an error (“[E] …”), the output given above is no real one.

In case you forgot to specify a required parameter, the output would instead look like:

[E] Missing runlimit specification – Use --time=<[hh:]mm>

If sbatch <jobscript> complains about missing (mandatory) parameters, even though all these seem to be defined using

#SBATCH …

pragmas, this may be caused by Windows linefeeds , which will not considered to be valid on UNIX/Linux.

If you wrote your job script on your windows PC/laptop and transferred it with scp to a login node, simply transform it with

dos2unix jobscriptfile

to a valid UNIX/Linux text file.

If above errors remain even after that, check all minus/hyphen characters. Here, you could have introduced dashes (long) or em-dashes (even longer), which do not work as “begin of a parameter” sign.

Before submitting jobs, you need to determine how many CPUs (= cores) you want (best) to use, how much main memory your scientific program will need and how long the calculating will take.

If your scientific program is already used in your group for problems like yours, you can ask your colleagues about their lessons learned.

If you start afresh with a new scientific program package or a new class of problems: prepare a comparably small test case (not more than 30 minutes runtime), and run it on one of the login nodes (with the desired number of cores) under the control of the UNIX “time” command as follows:

/bin/time --format='MaxMem: %Mkb, WCT: %E' myProgram <testcase>

After the run, you get for example

MaxMem: 942080kb, WCT: 1:16.00

on your STDERR channel.

After dividing “MaxMem” by 1024 (to get MBytes), you can determine your #SBATCH --mem-per-cpu= for that test case as

MaxMem in MByte
----------------- (plus a safety margin)
# of cores used

Your #SBATCH -t d-hh:mm:ss is then the “WCT” from above (plus a safety margin).

In our example and if you have used 4 cores:

 942080 / 1024 / 4 =
--mem-per-cpu=230

When you have run your test case with 2, 4, 8 and 16 CPU cores, you can roughly guess the scalability of your problem, and you can size your real job runs accordingly.

For running a lot of similar jobs, we strongly discourage from fiddling with shell script loops around sbatch / squeue. For any amount of jobs >20, use Slurm's Job Array feature instead.

Using job arrays not only relieves the Slurm scheduler from unnecessary overhead, but allows you to submit much more ArrayTasks than distinct jobs!

Example use cases are:

the same program, the same parameters, but lots of different input files
the same program, the same input file, but lots of different parameter sets
a serial program (unable to utilize multiple cores [multi-threading] or even several nodes [MPI]), but a lot of input files to analyze, and none of the analyses depends on results of any other, i.e. High-Throughput Computing

Rename the “numerous” parts of your job with consecutive numbering, eg. image1.png, image2.png or paramFile1.conf, paramFile2.conf etc.

Let's say you have 3124 sets, then set up a job script with

_______________

#SBATCH -a 1-3124
myProgram image$SLURM_ARRAY_TASK_ID.png > image$SLURM_ARRAY_TASK_ID.png.out

_______________

and submit it via sbatch. Slurm will now start one job with 3124 ArrayTasks, each one reading its own input image and writing to its own output file.

Even for the output/error file names specified with “-o” and “-e”, you can use "%. placeholders provided by Slurm:

#SBATCH -o /path/to/my/dir/out_%A_%a.txt
#SBATCH -e /path/to/my/dir/err_%A_%a.txt

^{%A = $SLURM_ARRAY_JOB_ID (the parent job id – same for all array tasks)}
^{%a = $SLURM_ARRAY_TASK_ID (the distinct counting number of each individual array task)}

If you need to limit the number of parallel running ArrayTasks, use

#SBATCH -a 1-3124%10

Slurm will then run at most 10 tasks concurrently.

Further details can be found in 'man sbatch' under “--array=”, or on the Slurm website.

Many scientific programs allow you to specify the amount of CPU cores to use, or the amount of GPU cards to use.

Instead of editing the command line of your scientific program (or its input files) over and over again, just to adapt it to what you request in # of CPU cores or # of GPUs, you can make use of Slurm variables set at job allocation (“starting the job”).

If for example your program is multi-threaded and uses OpenMP, you can write your job script as

#!/bin/bash

…

#SBATCH -c ##

export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK

/path/to/my/scientificProgram …

This way, it will always use exactly the amount of CPU cores you requested with “-c ##”.

Likewise, when using GPUs and your program's documentation explains to set eg. “--num-devices=#” to tell how many GPUs to use, you can specify

#!/bin/bash

…

#SBATCH --gres=#

/path/to/my/scientificProgram --num-devices=$SLURM_GPUS_ON_NODE …

This way, you just need to adapt the #SBATCH lines with “-c ##” or “--gres=#”, and your programs automatically “inherit” the exact settings.

_{(Please don't take # literally – replace it by the desired number of resources.)}

Use the “--multi-prog” option of srun, allowing you to run distinct and different programs from within one allocation (=job).

 #!/bin/bash

 #SBATCH …

 module purge

 module load mod1 mod2

 srun … --multi-prog myCtrlWorker.conf

The file myCtrlWorker.conf needs to follow the scheme

 N^o /path/to/program <parameter>

and contains per line the definition of which MPI rank (“N^o” ⇦ #SBATCH --ntasks=XX) exactly does what, and with which parameters.

In case the “worker” instances of your program need to know their own number, you may use the placeholders “%t” or “%o”, respectively (see ' man srun' under MULTIPLE PROGRAM CONFIGURATION).

Example

… for one controller controlling 24 workers:

 # MPI rank, then path to binary

 1 /path/to/controller_program <parameter for controller>

 2-25 /path/to/worker_program <parameter for worker, eg. WorkerID=%o>

With these two effective lines in myCtrlWorker.conf (the first one is a comment), the srun will first (1) start the controller, followed by 24 separate (2-25) “worker” instances, learning their own number by the %o placeholder.

Example

… for an arbitrary number of workers (only determined by #SBATCH --ntasks=XX):

 # MPI rank, then path to binary

 1 /path/to/controller_program <parameter for controller>

 * /path/to/worker_program <parameter for worker, eg. WorkerID=%o>

The * is the wildcard for “anything from 2 onwards”.

For how exactly these controller and worker processes “find and talk to each other” in main memory and which parameters this requires, please see your program's documentation.

Pending Jobs

Running Jobs

Check whether all directories mentioned in your job script are in fact there and writable for you.

In particular, the directory specified with

#SBATCH -e /path/to/error/directory/%j.err

for the STDERR of your jobs needs to exist beforehand and must be writable for you.

SLURM ends the job immediately if it is unable to write the error file (caused by a missing target directory).

Due to being a “chicken and egg” problem, a construct inside the job script like

#SBATCH -e /path/to/error/directory/%j.err
mkdir -p /path/to/error/directory/

cannot work either, since for Slurm, the “mkdir” command is already part of the job. Thus, any of “mkdir”s potential output (STDOUT or STDERR ) would have to be written to a directory which at begin of the job does not yet exist.

Make sure the relevant modules are loaded in your job script.

While you can load those modules right when logging in on the login node, since these are inherited by “sbatch myJobScript”, this is not reliable. Instead, it renders your jobs dependent on what modules you happen to have loaded in your login session.

We thus recommend to begin each job script with

module purge
module load <each and every relevant module>
myScientificProgram …

to have exactly those modules loaded which are needed, and not more.

This also makes sure your job is reproducible later on, independently of what modules were loaded in your login session at submit time.

This ususally is caused by nested calls to either srun or mpirun within the same job. The second or “inner” instance of srun/mpirun tries to allocate the same resources as the “outer” one already did, and thus cannot complete.

1. Script instead of Binary:

If you have

srun /path/to/myScientificProgram

in your job script, check whether “/path/to/myScientificProgram” in fact is an MPI-capable binary. Then, the above syntax is correct.

But if myScientificProgram turns out to be a script, calling srun or mpirun by itself, then remove the srun in front of myScientificProgram and run it directly.

2. Allocation per #SBATCH and per 'srun':

Such inadvertent “loop” is also a job script with two allocations: the “outer” one per #SBATCH -n 16 and the “inner” one with srun -n 16:

#SBATCH -n 16

…

srun -n 16 … /path/to/myScientificProgram

In that case, simply remove the “-n 16” from the 'srun' command. The notable advantage of srun in contrast to mpirun is precisely its ability to know from Slurm all about the job, and to “inherit” everything right away.

Example of such error:

srun: Job XXX step creation temporarily disabled, retrying
srun: error: Unable to create step for job XXX: Job/step already completing or completed
srun: Job step aborted: Waiting up to 32 seconds for job step to finish.
slurmstepd: error: *** STEP XXX.0 ON hpb0560 CANCELLED AT 2020-01-08T14:53:33 DUE TO TIME LIMIT ***
slurmstepd: error: *** JOB XXX ON hpb0560 CANCELLED AT 2020-01-08T14:53:33 DUE TO TIME LIMIT ***

Under SLURM, OpenMPI has an issue with respect to this MPI_Comm_spawn() routine (to later start further MPI ranks): it does not work.

If your program absolutely requires MPI_Comm_spawn(), you can just try to switch to another MPI implementation like “intelmpi”.

Example:

[mpsc0111:1840264] *** An error occurred in MPI_Comm_spawn

[mpsc0111:1840264] *** reported by process [2377252864,38]

[mpsc0111:1840264] *** on communicator MPI_COMM_SELF

[mpsc0111:1840264] *** MPI_ERR_SPAWN: could not spawn processes

[mpsc0111:1840264] *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,

[mpsc0111:1840264] *** and potentially your MPI job)

There is no magic by which Slurm could know the really important part or line of your job script. The only way for Slurm to detect success or failure is the exit code of your job script, not the real success or failure of any program or command within it.

The exit code of well-written programs is zero in case everything went well, and >0 if an error has occurred.

A script's exit code is the exit code of its last command.

Imagine the following job script:

#!/bin/bash
#SBATCH …
myScientificProgram …

Here, the last command executed is in fact your scientific program, so the whole job script exits with the exit code (or status) of “myScientificProgram” as desired. Thus, Slurm will assign COMPLETED if “myScientificProgram” has had an exit code of 0, and will assign FAILED if not.

If you issue just one simple command after “myScientificProgram”, this will overwrite the exit code of “myScientificProgram” with its own:

#!/bin/bash
#SBATCH …
myScientificProgram …
echo “Job finished”

Now, the “echo” command's exit code will become the whole job's exit code, since it is the last command of the job script. If the “echo” command succeeds (as it most certainly does), Slurm will assign COMPLETED even though “myScientificProgram” might have failed – the “echo”'s success covers up the failure of “myScientificProgram”.

To avoid that, save the exit code of your important program before executing any additional commands:

#!/bin/bash
#SBATCH …
myScientificProgram …
EXITCODE=$?
/any/other/job/closure/cleanup/commands …
echo “Job finished”
exit $EXITCODE

Immediately after executing “myScientificProgram”, its exit code is saved to $EXITCODE, and with its last line, your job script can now re-set this exit code (the one of your real “payload”).
That way, Slurm gets the “real” exit code of “myScientificProgram”, not just the one of the command which happens to be the last line in your job script, and will set COMPLETED or FAILED appropriately.

If you want to keep an eye on the real runtimes of your jobs, you can record this with just a one-liner to your STDERR channel (written to the file specified with “#SBATCH -e …”):

-----------------------

#SBATCH …
/usr/bin/date +“Job ${SLURM_JOBID} START: %F_%T.%3N” >&2
module purge
module load module1 module2 module3 …
/path/to/my/scientific/program …
EXITCODE=$?
/usr/bin/date +“Job ${SLURM_JOBID} END: %F_%T.%3N” >&2
exit ${EXITCODE}

-----------------------

Without any further ado like extra “echo” lines, the “date” command tells you the exact start and end times of your “payload”.

^{For further details on the % placeholders for the different date and time components please consult the 'man date' page unter “FORMAT controls”.}

Miscellaneous

If you want to be notified about status changes of your job(s) by mail, you write

#SBATCH --mail-type=…

in your job script and set … to BEGIN or END or FAIL or ALL, respectively. The scheduler will then send the respective mails to you.

Since all jobs are inextricably linked to your login name, ie. your TU-ID, we have configured Slurm to always send these mails to your current TU-ID mail address. We do not store it somewhere, instead Slurm asks the central IDM system of the TU.

So even if you change the mail address associated with your TU-ID, all scheduler mails will instantaneously go to the (new) target address.

Frequently Asked Questions – Software

Installation

Licenses

Runtime Issues

Remove all unnecessary modules. A job script should always start with

module purge
module load <only modules really required for this job>

to ensure a clean environment:.

Remember: whenever you load modules while you are on a login node, any job submitted from this modified environment will inherit these modules' settings!
Therefore, it is strongly recommended to use the above purging/loading statement in all job scripts.

Next, scrutinize your program's runtime libraries (“shared objects”) with

ldd -v /path/to/binary

Your $LD_LIBRARY_PATH might contain an unwanted directory, causing your program to load wrong or outdated libraries, which in fact should rather be coming from the modules you have loaded.

Particularly, the infamous“Bus error” can be caused by non-matching arguments or return values between calling binary and called library, thus causing “unaligned” memory access and crashes.

Check input and parameter files for wrong DOS/Windows line end characters . The Linux version of a program might not be ablce to cope with Windows CR/LF as the same program's Windows version happily can.

Frequently Asked Questions – Access

User Account

Projects

Frequently Asked Questions – batch scheduling system

Preparing Jobs

Pending Jobs

Running Jobs

Miscellaneous

Frequently Asked Questions – Software

Installation

Licenses

Runtime Issues

Machine Architecture

Miscellaneous