Skip to content

Internals

MVCC and transactions

Concurrent access and isolation

KiteDB runs transactions under Multi-Version Concurrency Control (MVCC). Multiple readers can access the database simultaneously without blocking each other or writers, and write transactions run concurrently, with conflicts detected at commit.

These transactions run inside one process. A writable open takes an exclusive lock on the database file, so no other open, in this process or another, can use the file until it is closed (see Opening a database). v0.2.18 and earlier did not lock the file.

MVCC is on by default since 0.3.0 (before, it was opt-in). Opening with mvcc: false (Rust .mvcc(false), Python OpenOptions(mvcc=False)) is deprecated and will be removed in a later release. Without MVCC, write transactions run one at a time and transactions read the latest committed state instead of a snapshot. MVCC is runtime state only: the file format is the same in both modes, and a file can be opened in either.

Snapshot isolation

Each transaction sees a consistent snapshot of the database as it existed when the transaction started, plus its own writes. Other transactions' uncommitted changes, and commits made after it started, are invisible. Reads outside a transaction see the latest committed state.

Two overlapping transactions
T1 startssees v1
T2 startssees v1
T1 commitswrites v2
T2 still sees v1. T1's commit stays invisible to T2 for as long as T2 runs; transactions that start after the commit see v2.

Version chains

When data is modified while other transactions are active, KiteDB keeps old versions in a chain:

Version chain for node "alice"newest first
v3: age=32
commitTs=150
T3 sees this
startTs=155
v2: age=31
commitTs=120
T2 sees this
startTs=125
v1: age=30
commitTs=80
T1 sees this
startTs=85

Each transaction walks the chain from the newest version and reads the first one committed before it started.

Each chain is keyed by the full IDs of what it versions: the node, the edge (source, type, destination), or the property or label together with its owner.

v0.2.18 and earlier: edge, property, and label chains were keyed by several IDs packed into one 64-bit integer, with as few as 12 bits per ID, so unrelated entries could share a chain. For example, two edges whose source node IDs differ by 220 shared version history.

Visibility rules

A version is visible to transaction T if
1
version.commitTs < T.startTs

The version was committed before T started

or

2
version.txid == T.txid

T created this version itself (read-your-own-writes)

Reads walk the chain from newest to oldest and return the first visible version. If none is visible, the entity does not exist for this transaction.

Write conflicts

KiteDB uses first-committer-wins to handle conflicts:

First-committer-wins

Scenario: T1 and T2 both modify "alice"

T1starts at ts=100
T2starts at ts=105

T1 commits first (ts=110)

No conflict, so the commit succeeds.

T2 tries to commit

"alice" was modified after T2 started (110 > 105), so T2's commit fails with a conflict error and T2 is rolled back.

Conflicts are checked at commit time against every key the transaction read or wrote. To resolve one, retry T2 so it reads the new value.

The failed commit applies nothing and ends the transaction. Rust returns KiteError::Conflict { txid, keys }, Python raises ConflictError, and JavaScript throws an Error whose message reads Failed to commit: Transaction <id> conflict on keys: [...]. Transactions that touch different nodes and edges don't conflict, so they all commit.

TypeScript
// Handling conflictstry {  db.transaction((ctx) => {    const alice = ctx.get(user, 'alice');    if (!alice) return;    const update = ctx.update(user, 'alice');    update.setAll({ age: alice.age + 1 });    update.execute();  });} catch (e) {  // The commit failed, for example because another  // transaction modified alice first. Nothing was applied;  // retry with fresh data.}

Lazy version chains

Version chains are only created when necessary: a commit records version history only while other transactions are open. Otherwise, modifications happen in place without versioning overhead.

Lazy versioning

When T1 commits a change to "alice":

No other active transactions

Modify in place, with no version chain

Other transactions active

Add a version to the chain and keep the old value

Serial workloads skip version-chain bookkeeping. Concurrent workloads still get snapshot isolation.

Bulk loads

A bulk load (beginBulk()) runs alone among writers: it waits for the open write transactions to finish, and write transactions that begin while it is open, or while it waits, wait for it. With no writer beside it, it records nothing for conflict checks. Readers never wait for it. Reads outside a transaction see the whole load once it commits, and a read transaction that began before the commit does not see it: when read transactions are open at the commit, it records version history for them, which takes time proportional to the size of the load.

v0.2.18 and earlier: beginBulk() failed with MVCC on ("bulk load requires MVCC disabled"), and the Python and TypeScript bulk helpers fell back to normal transactions.

Garbage collection

Old versions are cleaned up when no transaction can see them:

Version cleanup
  1. 1Compute the GC horizon: the start of the oldest active transaction, or the retention window (mvccRetentionMs, default 0: keep no history beyond what open transactions need), whichever is older
  2. 2
    For each version chain:
    Keep versions committed after the horizon, plus the newest one committed before it
    Prune older versions; no transaction can see them
  3. 3Truncate chains deeper than mvccMaxChainDepth (default 10), but never past the version the oldest active transaction reads

Runs:

On a writable open
Periodically in a background thread, one per writable open database (mvccGcIntervalMs, default 5 s). Read-only opens commit nothing, so they start no GC thread. Closing or dropping the database stops it without waiting for the next run

Note: long-running transactions hold back the horizon, so old versions stay in memory until they finish.

Options

Open options, named mvccGcIntervalMs in JavaScript and mvcc_gc_interval_ms in Rust and Python (likewise for the others):

OptionDefaultEffect
mvcctrueSnapshot isolation and concurrent write transactions. false is deprecated
mvccGcIntervalMs5000Milliseconds between background GC runs
mvccRetentionMs0How long to keep version history beyond what open transactions need
mvccMaxChainDepth10Version chain depth that GC truncates to

Transaction API

TypeScript
// Explicit transaction (the API is synchronous)db.transaction((ctx) => {  const alice = ctx.get(user, 'alice');  if (!alice) return;  const update = ctx.update(user, 'alice');  update.setAll({ age: alice.age + 1 });  update.execute();  // Commits when the callback returns  // Rolls back if it throws});// Batch operations: batch() executes each builder// inside a single transactiondb.batch([  db.insert(user).values({ key: 'bob', name: 'Bob' }),  db.insert(user).values({ key: 'carol', name: 'Carol' }),]);// Without an explicit transaction, each operation commits on its own

Next steps