Background jobs

Understanding background work in LoonFS

As a namespace accumulates writes, maintenance work keeps metadata reads indexed and efficient, updates derived indexes, and deletes expired resources. Maintenance runners schedule jobs and each job reads from durable state and performs an isolated unit of work, off the hot-commit/WAL path.

By default, the reference server runs maintenance automatically for any namespaces it serves. (LoonFS currently has no global namespace listing, so an operator must explicitly assign namespaces to a maintenance host if they need constant coverage):

loonfs maintenance loop --namespaces project-a --namespaces project-b

Embedded CLI usage does not leave a background process running after a command exits. In this mode, maintenance can be explicitly kicked off with maintenance loop.

Compaction

Metadata maintenance flushes the visible WAL tail into immutable metadata tables and then performs one bounded reorganization of those tables. The default policy kicks off this job at 32 uncompacted WAL-tail segments. If compaction is neglected, writes are eventually rejected with maintenance_required once an unmaintained tail reaches 128 segments.

Use loonfs maintenance metadata to start a single pass or loonfs maintenance flush to flush the current WAL tail regardless of its length.

Indexing

Indexing is another background job type that can be used to answer queries outside the scope of the core metadata tables. For example, one native index in LoonFS is the content index behind the grep-like Search capability.

Native LoonFS indexes can be complemented with external indexes. In many cases, Checkpoints and the Changefeed may be a more natural API for building non-native indexes.

Garbage collection

LoonFS uses append-only immutable objects and tombstone soft-deletion. Garbage collection later does mark-and-sweep passes over objects that are no longer reachable, and respects checkpoints, forks, upload receipts, and GC safety grace windows.