Skip to content

Storage

Also known as: Ceph filesystems (CephFS).

Overview

The following filesystems are generally accessible from all nodes in the cluster:

  • /home
    • home directory, max. 250GB and 1M files / user, SSD
    • best for code and configuration
    • avoid intensive inputs/outputs during the job
  • /local
    • long-term storage, slow and large (~2PB total), HDD
    • best for large-ish files that need to be accessed rarely, and can be accessed sequentially
    • avoid creating or reading millions of small (<1MB) files here
  • /cms/store
    • backed by the same physical disks as /local, so the same characteristics apply
    • can be read by anyone, requires grid certificate for writing and deletion
    • meant for storing CMS datasets and output files of grid jobs
  • /scratch/local
    • storage during job runtime only, fast and ephemeral, SSD
    • best for temporary outputs while the job is running
    • cleaned regularly (you should delete your outputs at the end of the job)
  • /scratch/persistent
    • medium term fast storage, up to ~1TB shared between all projects, SSD
    • best for jobs that require high-speed inputs that need to be accessed randomly
    • has rw access from the login node, ro access from the compute nodes

To get an overview of your disk usage, simply run:

show_storage

Example output:

Environment     Path                      Used(/quota)    Files[files,dirs](quota)  F/D-ratio
==============  ==============            ==============  ==============            ==============
-               /home/user                103GB(250GB)    308K[252K,56K](1.0M)      .0044
-               /local/user               237TB           200K[168K,33K]            .0050
-               /cms/store/user/cms_user  37TB            440K[430K,10K]            .0429

Notes
File to dir ratio should be between 0.1-1

Accessing CephFS

CephFS supports the standard POSIX commands (cd, ls, du, find, etc). However, when gathering statistics about directories, it may be faster to query the file system directly with the getfattr command:

getfattr -n <attribute> <directory>

where the <attribute> can be any of the following:

  • ceph.dir.files -- number of files, including hidden;
  • ceph.dir.subdirs -- number of subdirectories, including hidden;
  • ceph.dir.entries -- sum of ceph.dir.files and ceph.dir.subdirs;
  • ceph.dir.rfiles -- same as ceph.dir.files, but recursively summed;
  • ceph.dir.rsubdirs -- same as ceph.dir.subdirs, but recursively summed;
  • ceph.dir.rentries -- sum of ceph.dir.rfiles and ceph.dir.rsubdirs;
  • ceph.dir.rbytes -- total number of bytes allocated by the directory;
  • ceph.quota.max_files -- maximum number of files allowed in the directory;
  • ceph.quota.max_bytes -- maximum number of bytes allowed for the directory.

For example, to count the total number of files in your home directory, you can run the following:

getfattr -n ceph.dir.rfiles ~

Note

To get rid of the message getfattr: Removing leading '/' from absolute path names, you can either redirect it into /dev/null by appending 2>/dev/null to the command, or use the --absolute-names option of getfattr.

In case you want to print out the total number of files in each subdirectory that is located in your home directory, you can simply accomplish that with:

getfattr -n ceph.dir.rfiles ~/*/

To sort the directories by the number of files stored in each, you can do something along the lines of:

for d in ~/*/; do \
  echo $d $(getfattr --absolute-names --only-values -n ceph.dir.rfiles "$d"); \
done | sort -nr -k2

You can proceed similarly when estimating the space usage of each directory:

for d in ~/*/; do \
  echo $d $(getfattr --absolute-names --only-values -n ceph.dir.rbytes "$d"); \
done | \
sort -nr -k2 | \
awk '{cmd="numfmt --to=si --suffix=B <<< "$NF; cmd | getline $NF; close(cmd); print}'