CephFS vs RBD is a choice between two of the three ways into one Ceph cluster. Ceph stores everything as objects in its object store, RADOS, and three clients sit on top: the Ceph Block Device (RBD) presents block device images, the Ceph File System (CephFS) presents a POSIX file system that many clients mount, and the Ceph Object Gateway (RGW) presents S3- and Swift-compatible buckets over HTTP [3, 9]. The same cluster can serve all three at once [4]. Which one a workload uses depends on how many clients write the same data, whether its software needs a directory tree and which API it speaks.
All three stripe their data over many objects in the cluster [3]. They differ in what the client sees and what sits on the data path. CephFS clients read and write file data in RADOS directly, while Metadata Servers (MDS) keep directories and file attributes [3, 9]. RGW is an HTTP server between applications and the cluster, and it uses no Metadata Server [15].
CephFS vs RBD vs RGW at a glance
| RBD (block) | CephFS (file) | RGW (object) | |
|---|---|---|---|
| The client sees | A block device image [3] | A POSIX directory tree [9] | Buckets of objects over HTTP [3, 15] |
| Writers | One client writes an image at a time [5] | Many clients mount one tree [9, 11] | HTTP clients, each writing whole objects in the S3 API [15, 34] |
| Metadata | A file system inside the image, unknown to RBD [6] | MDS daemons, with their own pool [9] | Bucket index and log pools [20] |
| Access | Kernel module or librbd [4] | Kernel mount or FUSE [3, 28] | S3 and Swift REST APIs [15] |
| Kubernetes volumes that ceph-csi, Ceph's drivers for the Container Storage Interface (CSI), lists as generally available | File mode ReadWriteOnce, block mode ReadWriteOnce and ReadWriteMany [21] | File mode ReadWriteOnce and ReadWriteMany [21] | No ceph-csi volume type, applications use the REST APIs [15, 21] |
| Uses named by Ceph and Proxmox | Virtual machine and container disks [3, 25] | Shared home directories, HPC scratch, shared workflow storage [9] | Huge flat collections of files read and written whole [12] |
Block storage (RBD): one writer per image
What is RBD in Ceph? A Ceph block device is a thin-provisioned, resizable image striped over many storage daemons, with snapshots, replication and strong consistency from RADOS [4]. Virtualization is a common use: QEMU and KVM hosts typically use librbd to give guests a block device [3].
The exclusive-lock feature is on for newly created images and ensures that only a single client writes to an image at any given time [5]. It is used heavily in virtualization, where it keeps virtual machines from clobbering each other's writes [5]. By default it does not stop two clients from opening the same image and writing in turns: their writes get linearized as the lock passes back and forth, and the --exclusive mapping option turns those handoffs off [5]. An RBD image suits a disk that one machine writes.
RBD snapshots are read-only copies of an image at a point in time, and layering clones images such as virtual machine templates quickly [6]. Because RBD knows nothing of the file system inside an image, a snapshot is only crash-consistent unless the guest pauses I/O, for example with fsfreeze or the QEMU guest agent [6].
The iSCSI gateway exports RBD images as SCSI disks to clients without native Ceph support [8]. It has been in maintenance since November 2022, and Ceph advises new deployments to reach RBD images directly through librbd, QEMU or Kubernetes CSI, or through the NVMe-oF gateway [8].
File storage (CephFS): many clients, one namespace
CephFS is a POSIX-compliant file system on RADOS. Its metadata lives in a separate pool served by a resizable cluster of MDS daemons, and no gateway mediates data I/O [9]. Unlike many network file systems such as NFS, it keeps strong cache coherency across clients [11]. It relaxes a few POSIX rules: with simultaneous writers, a write that crosses object boundaries is not necessarily atomic, memory-mapped writes do not propagate coherently between hosts, and access times are not kept [11].
The metadata path is where CephFS costs time. A long listing stats every file and can be slow for large directories, a 10,000,000-file directory is less efficient than modest ones, and hard links carry a particular cost [12]. The MDS caches metadata, and performance differs greatly between workloads whose metadata fits in that cache and those whose metadata does not [12]. Each file system starts with one active MDS [13]. Workloads with many clients, perhaps in many separate directories, typically gain from extra active daemons, and a single application on a single client typically does not, unless it runs many metadata operations in parallel [13].
CephFS and NFS can work together: NFS-Ganesha exports CephFS namespaces and RGW buckets over NFS [14].
In Proxmox VE, RBD storage holds virtual machine disk images and container root disks, and CephFS storage holds files such as ISO images, container templates and snippets, with no disk images [25, 26].
Object storage (RGW): whole objects over HTTP
RGW is built on librados, keeps its own users, and serves an interface compatible with a large subset of the Amazon S3 API, plus Swift, in one shared namespace [15]. Ceph lists bucket versioning, multipart uploads, lifecycle rules, storage classes and bucket policies as supported [16].
The S3 API writes whole objects. Amazon's reference says S3 never adds partial objects, and changing one piece of metadata means putting the entire object again [34]. Ceph advises that an application keeping huge flat collections of files, reading and writing whole files at once, may be better off on the Object Gateway [12]. RGW can also export buckets over NFSv4, but only full, sequential writes work there, so editing a file in place fails [18].
Which workload uses which in a training cluster
Datasets, checkpoints and weights each set the choice, and the article on storage for model training covers their sizes and access patterns.
Datasets kept as files that are read and written whole match Ceph's advice for the Object Gateway [12], and a versioned bucket gives every object added its own version ID [17]. Software that expects a directory tree reads from CephFS, which many nodes mount at once [9, 21].
Checkpoints written by PyTorch's file-based distributed writer are one file per write request in a checkpoint directory, plus a global metadata file when ranks coordinate [35]. CephFS lets many nodes mount and write one directory tree, as a ReadWriteMany volume [21, 22]. The research on training on spot GPUs shows what checkpoint overhead and lost work cost.
Model weights kept by version fit versioned buckets, and object lock places a retention configuration, in governance or compliance mode, on a bucket [17]. Virtual machine disks and other volumes that one machine writes fit RBD [5, 7]. The comparison of self-hosted S3-compatible storage sets RGW against other object stores.
Ceph CSI: RBD vs CephFS in Kubernetes
Kubernetes v1.31 removed the CephFS and RBD volume plugins built into Kubernetes and recommends the CSI drivers [23]. The ceph-csi project lists as generally available an RBD volume in file mode for one node (ReadWriteOnce), RBD in raw block mode for one node or many, and CephFS for one node or many (ReadWriteMany) [21]. Kubernetes access modes do not enforce write protection once a volume is mounted [22].
CephFS vs RBD performance: what published tests show
The one published same-cluster comparison of two interfaces found here sets RBD against RGW. In two posts on Ceph's blog in March 2023, a Ceph developer tested pre-release Reef code on one 10-node cluster with 60 NVMe drives and 3x replication [29, 30]. RBD sustained about 71 GB/s of large reads and 25 GB/s of large writes [29]. RGW reached about 53 GB/s for 4 MB GETs and 18.6 GB/s for 4 MB PUTs [30]. The author traces much of the write gap to the data crossing the network twice, from client to RGW and from RGW to the storage daemons [30]. For CephFS, Ceph's documentation points to the metadata path and the MDS cache as the factors that set speed [12, 13].
Pools and hardware under each interface
Erasure-coded pools, which keep each object as data chunks plus coding chunks instead of whole copies, take full-object writes such as RGW's by default [19]. RBD and CephFS need overwrites enabled on the pool and keep their metadata in a replicated pool [19]. A CephFS file system needs at least two pools, one for data and one for metadata [10]. Ceph recommends at least three replicas and low-latency devices for the metadata pool, and a replicated default data pool, with any erasure-coded pool added for chosen directories [10].
MDS daemons are CPU-intensive and single-threaded, so they favor a high clock rate, and CephFS metadata pools and RGW index and log pools need SSDs at enterprise scale even when bulk data sits on hard drives [20]. The explainer on DGX, HGX and MGX shows which drives and network ports each kind of GPU system carries.
Rillor designs and builds distributed storage on Ceph for superintelligence (SI) work: block volumes, object buckets and shared file systems come from one cluster, and each workload reaches it in the form it needs.
Read the full report
The full report adds RBD's lock and snapshot rules, CephFS consistency and metadata, RGW's S3 features, Kubernetes access modes, erasure coding and hardware per interface, the test figures, survey results and all 36 sources.