Architecture overview

The shape of cetrafs: independent nodes, one shared disk, no network between them, and an on-disk layout built for compare-and-write.

Shared disk, not shared nothing

In a cetrafs cluster every node runs its own kernel, with its own page cache and no shared memory, and every node is attached to the same physical blocks. No data is replicated between nodes. There is one copy, on the LUN, and protecting it is the array’s job (RAID, snapshots, replication).

What the nodes have to agree on is who may change what. cetrafs settles that through the storage itself, never over the network.

The pieces

• The kernel module (cetrafs.ko) implements the filesystem: the namespace, file data, allocation, the cluster protocol, the journal, heartbeats and fencing, and the /proc interface.
• mkfs.cetra computes the geometry from the device size and writes every structure.
• fsck.cetra checks the volume offline in four passes and repairs only what has one correct answer.
• defrag.cetra asks the module to relocate a file into one contiguous run, online.

There is no daemon. The module does everything, and its only peer is the disk.

The on-disk layout

+----------------------------+
| Superblock                 |  block 0: every other region's offset is stored here
+----------------------------+
| Zone Group table           |  one block per Zone Group: owner node, free count
+----------------------------+
| Block Group table          |  one block per Block Group
+----------------------------+
| Global inode bitmap        |
+----------------------------+
| Global inode table         |  one inode record per 4 KiB block
+----------------------------+
| Node Registry              |  one 128-byte row per node, keyed by machine UUID
+----------------------------+
| Pending-free queues        |  one per Zone Group: frees handed to the owner
+----------------------------+
| Per-node journals          |  one private redo journal per registry row
+----------------------------+
| Block Group regions        |  header + data bitmap + data blocks, per Block Group
+----------------------------+

Nothing is found by position alone. Every region’s offset is a superblock field, and the mount checks the whole map (no overlap, no wrap, everything inside the device), not just the checksum. The file include/cetrafs.h in the source tree is the complete specification of the format; the module and the tools both compile against it. See the on-disk format reference.

Designed around a 512-byte atomic unit

The array performs compare-and-write atomically on one logical block, 512 bytes on a typical SAN LUN. cetrafs lays out its records to fit:

• Every record that nodes compete for (a Zone Group’s owner, a registry row, an inode’s lock fields) sits entirely in the first 512 bytes of its own block.
• One inode per 4 KiB block, even though an inode record is only about 200 bytes. With records packed together, one node’s write could overwrite a neighbouring record another node had just changed. One record per block makes that impossible by construction.
• Blocks are a fixed 4 KiB. A 512-byte block size was evaluated and rejected: it would mean roughly eight times as many metadata round trips and one directory entry per block.

Three problems, three mechanisms

When several nodes share one disk, three different things can go wrong, and cetrafs handles each with its own mechanism. All three are built on the same primitive, atomic compare-and-write.

Underneath them sit node liveness (cluster coordination), crash recovery (journaling), what happens when liveness gets it wrong (fencing), and integrity checking (checksums).

Last updated 21 October 2026