supercomputer:liger:gpfs
Table of Contents
Appropriate File System Usage
The following table gives an overview of the available file system resources on Liger Clusters.
Recommendation: ICI-SC has defined an environment variable which should be used as a base path for reading/writing large scratch files. Since the target of may change over time, it is recommended to use this variable instead of hard-coded paths.
Today NO BACKUP is performed neither on nor
Please do a copy by your own or ask your IT services.
A backup solution is under study for a futur deployment. Sorry for this inconvenience.
GPFS Filesystem
| File System | Usable Space | Limit | Backup | Description / Usage |
|---|---|---|---|---|
| 814 TB | no limit | NO | Full path to the user's standard scratch directory inside GPFS temporary storage location for applications with large size and I/O demands; data are automatically deleted (files after 90 days by modification and access date, empty directories after 3 days) |
| 30 TB | 5GB per user | NO1) | Full path to the user's home directory inside GPFS for source code, scripts, small static data, binaries, libraries, and applications |
/[group] | 45 TB | 100GB per project 1 millions files | NO2) | Full path to limited available user's project data directory inside GPFS storage for projects in collaboration with ICI; needed space must be applied for explicitly |
/softs | 2 TB | denied for users | NO3) | Software repository modules, flexlm, XF libraries, etc. |
Visibility
| Variable | Mount point | Access | Type | Protocol |
|---|---|---|---|---|
| /scratch/[login] | Login Compute+Visu | GPFS | device |
| /home/[login] | Login Compute+Visu | GPFS | NFS |
| /data/ | Login Compute+Visu | GPFS | NFS |
| not defined | /softs | Login Compute+Visu | GPFS | NFS |
More details about
- the different file systems; what file system to use for different data?
- the naming conventions and access right rules for ICI-SC file systems are given in HPC Data Rules for GPFS.
- the file system resources will be controlled by quota policy for each group/project. For more information see What data quotas do exist and how to list usage?
Good practice notes
(use tar for lot of small files, don't rename directories)
- Avoid a lot of small files
Numerousness small files should be reorganized within tar-archives to avoid long access times due to deficiencies in file processing of the underlaying operating system. - Avoid renaming of directories
a rename of directories within the data path should be done carefully because all data beyond the changed directory must be backed up once again. If a large amount of data is affected, it prevents backup of really new data in the entire file system and wastes system resources like CPU time.
supercomputer/liger/gpfs.txt · Last modified: 2016/04/21 23:10 by Richard.Randriatoamanana@ec-nantes.fr
Except where otherwise noted, content on this wiki is licensed under the following license: CC Attribution-Noncommercial-Share Alike 4.0 International
