CEPH / STORAGE DESIGN
Metadata, DB & WAL explained
Two different layers, two different planning decisions. A CephFS metadata pool organises the filesystem. BlueStore DB/WAL belongs to each individual OSD.
| Component | Purpose | Placement |
|---|---|---|
| CephFS metadata pool | Filesystem namespace: directories, names, permissions | A replicated RADOS pool across OSDs; coordinated by MDS services |
| BlueStore block.db | Internal OSD metadata in RocksDB | Local to an OSD; optionally on a separate faster device |
| BlueStore block.wal | Write-ahead log for internal operations | Local to an OSD; normally colocated with an external DB if no separate WAL is configured |
↓ MDS
Replicated metadata pool↓ RADOS / OSDs
The local storage layout underneath the pools.
A fast metadata pool still uses OSDs with their own DB/WAL. The two layers coexist.
What does the MDS do?
A CephFS Metadata Server coordinates filesystem metadata and caches active state in memory. Its persistent journal and metadata live in RADOS, not solely on the MDS server’s boot disk. File contents use data pools. An MDS standby helps with service failover; it does not replace protection of the metadata pool.
Where do DB and WAL live?
A BlueStore OSD can keep data, DB and WAL on one device. With HDD data storage, moving block.db to faster SSD/NVMe can help metadata I/O. Without a separate block.wal, WAL follows that DB device. WAL records internal updates to support consistent recovery after an interruption. It is not a general-purpose cache for every application write or a backup.
Example: four HDD OSDs, one shared NVMe
Imagine four HDD OSDs on one host, each with its DB allocation on one NVMe. Losing that NVMe can take all four dependent OSDs out of service even though the HDDs are intact. DB/WAL is essential state, not a disposable cache. Cluster redundancy and successful recovery must cover the combined failure.
This does not automatically mean cluster-wide data loss. With host-level placement, the loss stays within one host failure domain—but recovery has to handle all affected OSDs. Partitioning one SSD does not create independent physical failure domains.
Which bottleneck are you trying to solve?
Slow directory operations suggest examining MDS resources and metadata-pool latency. Slow OSD metadata operations suggest examining DB placement and utilisation. These are investigation paths, not diagnoses. Measure the real workload: faster DB/WAL does not automatically accelerate large sequential transfers or remove a network bottleneck.
Protect metadata as carefully as file contents
CephFS metadata requires a replicated pool; erasure coding is not supported for that pool. Ceph recommends at least three copies and fast, low-latency OSDs. The chosen CRUSH rule must separate copies across the intended failure domains. Intact file-data objects alone do not guarantee a usable filesystem after metadata loss.
Size each layer separately
There is no universal DB percentage for every release and workload. Consider object count, object size, omap usage, growth and compaction, then monitor DB utilisation and spillover. When an external DB fills, metadata can spill onto the slower data device. That may reduce performance; it is different from the DB device failing.
What belongs in the capacity calculator?
Enter only the data drives assigned to the pool you are modelling. Dedicated DB/WAL devices and a separate CephFS metadata pool need their own budget. For example, four DB allocations of 80 GB require 320 GB of assigned DB space before additional planning headroom. This illustrates allocation arithmetic, not a recommended DB size.
Open the capacity calculator →EURONAS / eEKAS
Separate choices, guided configuration
eEKAS presents DB/WAL placement and CephFS metadata-drive selection as separate configuration decisions. Match both to your workload and failure model.
Explore eEKAS at euroNAS ↗Technical sources
Check the documentation for your installed Ceph release; defaults and sizing guidance change.