You can use the following file systems:
|Access time||fast (dependent on overall file system traffic)|
|Files accessible||global for all nodes|
|Persistence||permanent||during the project's validity term||after 8 weeks, files will be deleted unconditionally and without further notice|
|Quota||40 GByte, more on request||on request only||10 TByte or 2Mio. files, more on request|
Snapshots (see below) +
daily tape backup (for desaster recovery only)
static input data, results of finished jobs, low-volume I/O
Do not use home, groups or projects for running jobs!
|running jobs' input/output, intermediary files (CPR), high-volume I/O|
Since the migration to the new storage system in October 2019, the global file systems do not differ in throughput or latency any longer.
Due to being too much volume for snapshots and backup: Do not use
projects for I/O of running jobs!
For this class of file systems with low turnover in terms of number and size of files, the following backup mechanisms are in place:
- a daily tape backup
This is mainly to protect against catastrophic damages to the file system, and restores are not directly available to users.
- the snapshot mechanism
For your own restore/recover purposes, the GPFS file system (see “Technology” below) does regular snapshots of the content. Available via the hidden folder “
.snapshots/” in every subdirectory, you can copy back what has been lost, ie. any file inadvertently deleted. For details, see “Technologies” below.
Do not use directories on these file systems for running jobs!
The home directory should be used for all files that are important and need to be stored permanently.
Every user can only store a small amount of data here: default quota is currently 40 GBytes. In well reasoned cases and on request, this quota can be increased. The folder /home/$USER (“Home”) is created with each user account. It is accessible by the environmental variable
/work/projects and /work/groups
Available on request, groups (institutes) can get a group folder, to share static input data and common software (versions) for their members and coworkers.
Likewise, projects with more than a few members can request a projects folder for the same purposes.
This class of file systems is explicitely optimized for high I/O volume and high throughput for jobs and applications.
Due to the high turnover in created/deleted/changed files, an (expensive) legacy tape backup would “explode” in volume and metadata. For the same reason, even the GPFS-internal snapshots could rather quickly overfill the file system.
There is absolutely no backup for the following file systems. What you delete here, is gone forever, and cannot be restored/recovered, not even by administrators.
Here, almost unlimited disk space is available for all users, but only for a limited time: After 8 weeks of not being written to nor read from, the files will be deleted unconditionally without further notice.
Standard quota is currently 10 TByte and 2 million files. In well reasoned cases and on request, this quota can be increased.
/work/scratch/$USER (“scratch”) is created with each user account. For eg. (job) scripts, it is accessible by the environmental variable
All shared, cluster-wide file systems above are based on IBM's Spectrum Scale (formerly known as General Parallel File System). This commercial product can share large disk arrays via Infiniband among thousands of nodes.
Of course, arbitrating read/write requests from such numbers of nodes to individual files will take some more time than accessing local disks. In addition, all running jobs and all logged-in users are equally working on this shared storage resource.
That's why you sometimes see (hopefully short) “hiccups” when doing a
ls -l or similar commands. This is completely normal and results from the principle “common, shared file system available everywhere”.
For folders on the low-volume I/O class of file systems, GPFS automatically creates periodic snapshots, allowing you to access (and restore) older versions of your files without assistance by the admins. Snapshots are saved to the hidden folder
.snapshots (you would not see this folder listed, not even by an '
ls -la'). Nonetheless, you can go to that hidden folder by explicitely “
cd .snapshots” (<TAB>-completion does not work either, you have to fully type
.snapshots). Being in
.snapshots/, you can do 'ls -l' and 'cd' as usual, and access former versions (or states) of all your data (within the
Files from the snapshot folder still occupy storage space! Therefore, it is possible your home folder's quota is exceeded, even though the '
df' command still shows less usage.
Snapshots cannot be deleted (deleting data creates copies of the snapshot).
Frequent saving and deleting files fills up the snapshot area and requires space at the containing folder. If possible, this should thus be avoided (so do not use home, groups or projects folders for high-volume I/O eg. for I/O of running jobs!).
In urgent cases, the snapshot folder can be deleted by the administrators.