Storage

Also known as: Ceph filesystems (CephFS).

Overview

The following filesystems are generally accessible from all nodes in the cluster:

• /home
  • home directory, max. 250GB and 1M files / user, SSD/NVME
  • best for code and configuration
  • avoid intensive inputs/outputs during the job
  • has read+write access from the login node, read+write access from the compute nodes
• /local
  • long-term storage, slow and large (~2PB total), HDD
  • best for large-ish files that need to be accessed rarely, and can be accessed sequentially
  • avoid dealing with millions of small (<1MB) files, or doing millions of small operations (e.g. log file writing)
  • has read+write access from the login node, read+write access from the compute nodes
• /cms/store
  • backed by the same physical disks as /local, so the same characteristics apply
  • can be read by anyone, requires grid certificate for writing and deletion
  • meant for storing CMS datasets and output files of grid jobs
• /scratch/local
  • storage during job runtime only, fast and ephemeral, SSD/NVME
  • best for temporary outputs (e.g. log files, job otuputs) while the job is running
  • the logical path /scratch/local may map to different underlying physical hardware on different nodes, so data cannot be shared across nodes
  • no guarantee of data storage beyond the lifetime of the job: you should delete your outputs at the end of the job
  • has read+write access from the compute nodes
• /scratch/persistent
  • medium term fast storage, up to a few tens of TB shared between all projects, SSD/NVME
  • best for jobs that require high-speed inputs that need to be accessed randomly, e.g. small sets of frequently accessed NANOAOD, ML input datasets
  • has read+write access from the login node, read-only access from the compute nodes

To get an overview of your disk usage, run:

show_storage

Example output:

Environment     Path                      Used(/quota)    Files[files,dirs](quota)  F/D-ratio
==============  ==============            ==============  ==============            ==============
-               /home/user                103GB(250GB)    308K[252K,56K](1.0M)      .0044
-               /local/user               237TB           200K[168K,33K]            .0050
-               /cms/store/user/cms_user  37TB            440K[430K,10K]            .0429

Notes
File to dir ratio should be between 0.1-1

Accessing CephFS

CephFS supports the standard POSIX commands (cd, ls, du, find, etc). However, when gathering statistics about directories, it may be faster to query the file system directly with the getfattr command:

getfattr -n <attribute> <directory>

where the <attribute> can be any of the following:

• ceph.dir.files -- number of files, including hidden;
• ceph.dir.subdirs -- number of subdirectories, including hidden;
• ceph.dir.entries -- sum of ceph.dir.files and ceph.dir.subdirs;
• ceph.dir.rfiles -- same as ceph.dir.files, but recursively summed;
• ceph.dir.rsubdirs -- same as ceph.dir.subdirs, but recursively summed;
• ceph.dir.rentries -- sum of ceph.dir.rfiles and ceph.dir.rsubdirs;
• ceph.dir.rbytes -- total number of bytes allocated by the directory;
• ceph.quota.max_files -- maximum number of files allowed in the directory;
• ceph.quota.max_bytes -- maximum number of bytes allowed for the directory.

For example, to count the total number of files in your home directory, you can run the following:

getfattr -n ceph.dir.rfiles ~

Note

To get rid of the message getfattr: Removing leading '/' from absolute path names, you can either redirect it into /dev/null by appending 2>/dev/null to the command, or use the --absolute-names option of getfattr.

In case you want to print out the total number of files in each subdirectory that is located in your home directory, you can accomplish that with:

getfattr -n ceph.dir.rfiles ~/*/

To sort the directories by the number of files stored in each, you can do something along the lines of:

for d in ~/*/; do \
  echo $d $(getfattr --absolute-names --only-values -n ceph.dir.rfiles "$d"); \
done | sort -nr -k2

You can proceed similarly when estimating the space usage of each directory:

for d in ~/*/; do \
  echo $d $(getfattr --absolute-names --only-values -n ceph.dir.rbytes "$d"); \
done | \
sort -nr -k2 | \
awk '{cmd="numfmt --to=si --suffix=B <<< "$NF; cmd | getline $NF; close(cmd); print}'