Skip to content

Guides

Concurrency

Parallel readers, concurrent writers, and MVCC

Within one process, many threads can read a KiteDB database at the same time, and write transactions on different threads run concurrently. Since 0.3.0, MVCC is on by default: each transaction reads a snapshot, and a commit that conflicts with a write committed since its transaction began fails, so you retry it.

Concurrency model

KiteDB uses multi-version concurrency control (MVCC):

  • Multiple concurrent readers – Any number of threads can read simultaneously, without waiting for open write transactions
  • Concurrent writers – Each thread runs its own transaction. Write transactions run at the same time; commits that arrive together are written as one group, each applied whole and in order
  • Conflicts at commit – A commit fails with a conflict error if a key its transaction read or wrote was changed by a commit since the transaction began, so concurrent read-modify-write transactions cannot lose updates. Nothing from the failed transaction is applied; retry it (see Isolation and conflicts). Writers that touch different nodes and edges don't conflict
  • Non-MVCC mode (deprecated) – With mvcc: false, a write transaction waits in begin until the open one commits or rolls back. It will be removed in a later release
TypeScript
// Reads are synchronous. Calls on one handle run one after// another on the JS thread, so there is nothing to await.const alice = db.get(user, 'alice');const bob = db.get(user, 'bob');const following = alice ? db.from(alice).out('follows').toArray() : [];// For parallel reads, open the file in each worker thread with// readOnly: true. Read-only handles share the file lock; a writable// handle holds it exclusively.const reader = await kite('./data.kitedb', {  nodes: [user],  edges: [follows],  readOnly: true,});

Concurrent writers by language

  • Rust – The high-level Kite takes &mut self for writes, so behind a lock its writers take turns. For concurrent write transactions, share a low-level SingleFileDB (from open_single_file) in an Arc; each thread calls begin and commit on it
  • Python – Threads can share one Database or Kite; each thread's begin opens its own transaction. Each call holds the GIL, except the ones that can block (begin, commit, checkpoints, backups and the like), which release it while they wait
  • Node.js – JS on the main thread runs one transaction at a time per handle, and a writable handle locks the file, so writes from JS don't run concurrently

Performance notes

Read throughput typically improves with parallel readers. Writer threads build their transactions in parallel, and commits that arrive together are written as one group: one WAL write, one header write and, with syncMode: 'Full', one fsync. See the parallel write scaling notes for measurements. For one-shot ingest, a bulk load (beginBulk()) through one writer is fastest. Measure with your workload and tune batch sizes and sync mode accordingly.

Best practices

  • Batch writes – Group multiple writes into single operations to pay the commit cost once
  • Use transactions for atomicity – Group related writes; readers see all of a commit or none of it
  • Retry on conflict – A commit that fails with a conflict applied nothing; run the transaction again. Short transactions conflict less often
  • Keep transactions short – While a transaction is open, commits keep version history for it in memory
  • Profile your workload – The optimal thread count depends on your read/write ratio and data access patterns

MVCC and transaction semantics

MVCC (multi-version concurrency control) is on by default since 0.3.0:

  • Multiple readers can run concurrently
  • A transaction reads the state as of its begin, plus its own writes, never a later commit. Reads outside a transaction see the latest committed state
  • Write transactions run concurrently, and write conflicts are detected at commit time
  • Commits are applied one at a time; a commit briefly blocks new reads while it publishes its changes
  • Each committed transaction is atomic
  • A background thread per writable open database prunes version history every mvccGcIntervalMs (5000 ms by default); read-only opens start none. Closing the database stops it without waiting for the next run

Opening with mvcc: false (Rust .mvcc(false), Python OpenOptions(mvcc=False)) turns MVCC off. That mode is deprecated and will be removed in a later release; it logs no warning. Without MVCC, write transactions run one at a time: a write begin waits until no other write transaction is open (in Python it releases the GIL while it waits), and transactions read the latest committed state instead of a snapshot. The file format is the same in both modes, so existing files open under the new default.

TypeScript
// Atomic transaction (auto-commit on success, rollback on error)await db.transaction(async (ctx) => {  const alice = ctx.get(user, 'alice');  if (alice) {    const update = ctx.update(user, 'alice');    update.set('name', 'Alice Updated');    update.execute();  }});

Limitations

  • Single-process only – Concurrent access is within a single process; multi-process access requires external coordination
  • Write serialization – Commits are applied one at a time (without MVCC, whole write transactions are). High-write workloads may see contention
  • Memory overhead – Version history uses additional memory while transactions are open, until garbage collection removes it

Next steps