cetrafs

Beta cetrafs 0.7.4 · Linux kernel module + tools

Many hosts, one LUN.
One filesystem.

cetrafs is a clustered, shared-disk filesystem for Linux. Several hypervisors mount the same iSCSI or Fibre Channel LUN at once and keep their VM disk images directly on it, coordinating through the storage itself. It is an open alternative to VMFS.

Try it in 15 minutes Read the docs

Shared-disk clustering

Up to 64 nodes mount one block device. Each node allocates only from the Zone Groups it owns, so two nodes never draw from the same free-space pool.

One atomic primitive

Every contested claim is a single SCSI COMPARE AND WRITE on one 512-byte record. It is the same idea as VMFS's ATS, with no DLM, no corosync and no network protocol in the data path.

Crash consistent

Each node has its own write-ahead redo journal. It is replayed at mount and whenever a dead peer is reclaimed, and a replay guard refuses to roll back blocks that someone else has since rewritten.

Checksummed end to end

CRC32C on every on-disk structure, verified by the kernel on read, with a retry for torn reads. Per-inode generations catch inode-number reuse across nodes.

Live migration

With nofilelock, a VM image can be open on two nodes during a Proxmox/qemu online-migration handoff, and the handoff stays coherent even across a snapshot.

Self-healing upkeep

Online defragmentation, a background scrubber that reclaims leaked blocks, and an fsck.cetra that repairs faults with one correct answer and refuses to guess at the rest.

Built for one workload

cetrafs is a VM datastore, not a general-purpose POSIX cluster filesystem. Keeping to that narrow contract is what lets it run without a cluster lock manager in the I/O path.

613–710
MB/s O_DIRECT writes on a 3PAR LUN, about 2.3× GFS2 on the same LUN
64
hosts per volume at most, each owning its own allocation regions
16 TiB
maximum size of a single file or VM image (depth-2 extent tree)
0
daemons, lock managers or cluster networks required in the data path

Find what you need

Pick a section based on what you are trying to do.

Tutorials · learning

Learn by doing

Guided lessons, from a throwaway loop-device volume to a Proxmox cluster that live-migrates VMs.

  • Your first cetrafs volume
  • A two-node cluster on a shared LUN
  • A Proxmox VE shared datastore
How-to guides · tasks

Get a job done

Short recipes for operators: install, probe storage, tune for multipath, repair, recover.

  • Check a LUN for atomic CAW
  • Set the stale window for multipath
  • Recover from a node failure
Concepts · understanding

Understand the design

Why cetrafs works the way it does: coordination, locking, journaling, fencing, and how it compares to VMFS and GFS2.

  • Cluster coordination
  • Fencing and split-brain
  • cetrafs vs GFS2
Reference · information

Look it up

Exact facts: tool flags and exit codes, mount options, procfs fields, module parameters, limits, the on-disk format.

  • mkfs.cetra
  • /proc/fs/cetrafs
  • Limits and unsupported operations

Young software. Honest about it.

cetrafs has been reviewed, fuzzed and crash-tested heavily, but it is beta software and has not had years of production use the way VMFS and ext4 have. Try it on storage you can afford to reformat, and keep independent backups.

Project status Known limits