Internals
Single-file format
The .kitedb file layout
KiteDB stores a database in one .kitedb file containing a header, a write-ahead log, and a snapshot. The log is a fixed WAL area plus, when it fills, WAL segments elsewhere in the file. One file is easy to copy and deploy, and the header records where the current snapshot, the WAL and the WAL segments are.
File layout
The header
The header holds the metadata needed to open the database. KiteDB keeps two copies of it, in pages 0 and 1 (4 KB each at the default page size). Each header update is written to the copy that is not current, so the current copy is never overwritten. On open, KiteDB checks both copies and uses the valid one with the higher change counter. If a crash tears a header write, that copy fails its checksum and the other copy is used.
- Magic bytes
- "KiteDB format 2\0"16 bytes. Releases v0.2.3 to v0.2.18 accept only "KiteDB format 1\0" (earlier ones only "RayDB format 1"), so they refuse these files instead of misreading them. This version still reads that magic, and a writable open rewrites both slots in the new one as its last step, so an open that fails leaves them in the old one; a crash between the two writes can leave page 0 in the old magic, which v0.2.18 then opens as the file was before that open, until the next writable open finishes. It refuses files of v0.1.4 to v0.2.2 ("RayDB format 1"), as releases have since v0.2.3
- Versions
- Format version, minimum reader version, and feature flags. A header that names WAL segments is version 3 with minimum reader version 3, so a build that reads version 2 but not segments refuses the file instead of missing commits; otherwise it is version 2. Releases up to v0.2.18 read none of this (they check only the magic and a checksum, and read only the first header page): the magic keeps them out. Open refuses a file that needs a newer reader or has flags it does not implement, and opens a newer format read-only
- Page size
- 4096default
- Change counter
- Incremented on every header write; open uses the higher one
- Snapshot location
- Start page, page count
- Snapshot generation
- Incremented on each checkpoint
- WAL location
- Start page, page count
- WAL pointers
- Head and tail, plus each WAL region's head and the active region
- WAL segment table
- Up to 64 entries, each with a seq, start page, page count, sealed flag and byte length; plus covered, the newest segment seq the snapshot covers, and the next seq
- Checkpoint flag
- Set by an earlier version's background checkpoint between its cut and its install; this version never sets it
- WAL salts
- One per WAL region, mixed into each record's checksum and replaced whenever the WAL starts over
- Counters
- Max node ID, next transaction ID, last commit timestamp
- Checksums
- CRC-32 over the header fields, and one over every byte of the page but the two checksums, so a page torn between two writes, at any 512-byte sector, fails it. A header in the old magic keeps its old page checksum (over the header fields' checksum too, which leaves the fields out of it); one of those that names WAL segments, which only unreleased builds wrote, is refused with its file
Atomic updates
A checkpoint writes the new snapshot to a different region than the current one and then switches the header to it. Blocking and background checkpoints follow the same order; the WAL page covers how background checkpoints handle writes that arrive in the meantime.
- 1
Build the new snapshot in memory from the current snapshot and the changes it covers
A blocking checkpoint uses the live delta; a background checkpoint replays the WAL segments up to its cut
- 2
Write it to pages no header names
The first free range that holds it, otherwise the end of the file. The current snapshot's pages are never overwritten.
- 3
fsync()so the snapshot is durable - 4
Write a header that points to the new snapshot, and to the log it does not cover, into the inactive header page
- 5new snapshot is current
fsync()the header - 6
Write the same header into the other page and
fsync()Now neither header page points to the old snapshot
- 7
Free the old snapshot's pages and the WAL segments the new header drops
Later snapshots and segments can reuse them
If the process crashes:
A header write torn by the crash fails its checksum, and open uses the other header page. No header names the pages a crashed checkpoint wrote, and open frees them.
WAL area
The WAL area is a fixed-size, append-only log. The default size is 4 MB. The size is fixed when the file is created; change it with resizeWal (offline) or rebuild into a new file.
A full WAL spills into a WAL segment instead of forcing a checkpoint, so the WAL's size no longer limits a transaction. Automatic checkpoints start once the log the snapshot does not cover reaches checkpointLogRatio (default 0.5) times the snapshot's size, at least three eighths of the WAL (where earlier releases checkpointed), at most checkpointLogBudget (default 128 MiB); checkpointThreshold is deprecated and has no effect. So the WAL's size still sets when a small database checkpoints, and the floors of the segment limit (16 WALs) and the segment extent (2 WALs); a larger WAL also means fewer spills. The WAL page covers why spills and background checkpoints are crash safe.
When a commit does not fit in the WAL, the WAL spills:
walSegmentLimitThe secondary region is only read, when opening a file from an earlier version, whose background checkpoints wrote there. A writable open moves those records back to the primary region, or into a WAL segment if they don't fit.
Snapshot area
The snapshot area holds the graph data in CSR format:
- Node ID mappings
- Physical ↔ logical ID translation
- Out-edge CSR
- offsets[], destinations[], edge_types[]
- In-edge CSR
- offsets[], sources[], edge_types[]
- Properties
- Node and edge property values
- String table
- Deduplicated string storage
- Key index
- Hash-bucketed node key lookups
- Schema
- Labels, edge types, property keys
Each section is compressed independently with zstd. Typical size: 40–60% of the raw data.
File growth
The header and WAL area have fixed sizes; the snapshot grows with your data, and WAL segments come and go with the log:
These examples assume the default 4 MB WAL and a checkpoint that covered the whole log, so the file holds no WAL segments, as after a clean close. Until a checkpoint covers them, WAL segments add to these sizes, up to walSegmentLimit.
Because a checkpoint never overwrites the current snapshot, the file can hold an old snapshot's pages next to the current one, with WAL segments between them. Snapshots and segment extents go into the first free range that holds them, else the end of the file, and free pages at the end of the file are truncated after a checkpoint. Open treats every page no header names as free. A clean close checkpoints the WAL segments away, then moves the snapshot down next to the WAL when free pages lie in front of it and truncates the file, as vacuum does. A file closed with a transaction open, or whose close-time checkpoint failed (close warns and goes on), keeps its segments where they are and is not compacted, as is a database dropped without closing: the next open replays them.
Single-file vs multi-file
KiteDB previously supported a directory-based format. Single-file is now the default:
| Aspect | Single-file | Directory (legacy) |
|---|---|---|
| Portability | Copy one file | Copy entire directory |
| Atomic ops | Header flip | Manifest + renames |
| Disk usage | ~40% smaller | More overhead |
| Complexity | Simpler | More moving parts |
Opening a database
- 1
Lock the file
Exclusive for a writable open, shared for a read-only open
- 2
Read both header pages and use the newest one that passes its checksums
A writable open converts a file in the old single-header layout first
- 3
If an earlier version left records in the secondary WAL region, move them back
A format version 2 background checkpoint wrote there between its cut and its install. A writable open moves those records after the primary region's, or into a WAL segment if they don't fit. Read-only opens replay both regions in place without writing.
- 4
Writable opens: free every page no header names
Old snapshots' pages, and extents a crash left before a header named them
- 5
mmap()the snapshot area - 6
Parse snapshot sections
- 7
Replay committed transactions from the WAL segments and the WAL, in commit order, to rebuild the delta
Only those whose Commit record lies after the last segment the snapshot covers
Ready for queries
Incomplete transactions found during replay are discarded, since they never committed. Recovery runs automatically on open.
The file lock lets one writer or any number of readers use a file at a time. A writable open fails if another open of the same file, in this process or another, holds any lock on it. A read-only open fails only if a writer holds it.
A read-only open asks the operating system for read access only and never writes to the file. Replay of the WAL segments and the WAL rebuilds the delta in memory, and closing the database leaves the file untouched. In a file where an earlier version left records in the secondary WAL region, a read-only open replays both regions in place; the next writable open moves the records back.
Next steps
- WAL and durability: how the write-ahead log provides crash safety
- Snapshot and delta: how reads merge these two sources