Performance

zetl is designed for large vaults — tens of thousands of notes — without sacrificing startup speed or interactive responsiveness.

(given fast-startup)

Targets

Metric	Target	Measured (10k vault)
Cold-start full scan	< 200 ms	140 ms
Incremental re-index (1 file)	< 50 ms	8 ms
Graph query p99	< 10 ms	4 ms
Search (10k files)	< 2 sec	—
Reasoning (10k rules)	< 500 ms	—
TUI startup	< 200 ms	—
View startup	< 200 ms	—
Scroll frame time	< 16 ms	—
Merkle construction overhead	< 20%	—
Diff (2k pages, 50 changed)	< 500 ms	—
Watch event latency (p95)	< 500 ms	—

Strategies

Incremental parsing

The Scanner only re-parses files whose mtime has advanced since the last run. On typical editing sessions (1–5 changed files), this cuts scan time by 99% compared to a full rebuild. See architecture/Cache.

Merkle-based change detection

BLAKE3 content hashes let zetl skip unchanged files even when mtimes are unreliable (e.g., after git checkout). See Merkle Tree.

Memory layout

The Link Graph is stored as a flat adjacency list rather than a pointer-linked structure. This improves cache-line utilisation during multi-hop traversal.

Index-free search

Search Command uses direct file scanning rather than an inverted index. This trades query speed on very large vaults for zero build-time indexing cost. See ADR-002 Search Without Index.

See also: architecture/Cache, Scanner, Merkle Tree, Link Graph

Backlinks

Indexline 40
Demo Vaultline 9
Scannerline 47
Link Graphline 34
ADR-012 Changed Files Reconstructionline 32
ADR-002 Search Without Indexline 30
ADR-002 Search Without Indexline 17

Linked Pages

Link Graph

zetl builds a directed graph where nodes are pages and edges are concepts/Wikilinks. The graph powers all link-based queries.

(given directed-graph)
(given multi-hop-traversal)

Data model

Node — a Markdown page, identified by its title (filename without .md, case-insensitive)
Edge — a wikilink from one page to another, with optional alias, heading, or block reference

The graph is built by the Scanner and cached by architecture/Cache for incremental re-scans.

Resolution

Page names are resolved case-insensitively. When a wikilink target like [[cache]] appears, zetl matches it to Cache.md regardless of casing. For disambiguation between pages with the same name in different directories, use path-qualified links like [[architecture/Cache]].

Queries

Command	Description
`links`	Forward links from a page, with configurable depth
`backlinks`	Pages linking to a target, with depth traversal
`path`	Shortest link path between any two pages
`export`	Full graph as JSON

These are exposed through Links Command, Path Command, and List and Export Commands. With the --with-conclusions flag, link results are cross-referenced with the Reasoning Engine to show what each linked page contributes to the vault’s logic.

Library

Built on petgraph, a Rust graph data structure library. The graph is stored as a flat adjacency list for cache-line efficiency during multi-hop traversal. See Performance.

Merkle Tree

zetl builds a content-addressed Merkle tree over each Markdown file during indexing. This enables two-tier cache invalidation, SPL grounding, and Drift Detection.

(given content-addressed-hashing)

Structure

Each file produces a flat list of Merkle leaf nodes, computed from pulldown-cmark AST block-level elements:

Leaf type	Source
Heading	`## Section Title`
Paragraph	Prose text blocks
SplBlock	```spl fenced code blocks
Code	Non-SPL fenced code blocks
Table	Markdown tables
List	Ordered/unordered lists
Blockquote	`>` block quotes
Frontmatter	YAML between `---` fences

BLAKE3 hashing

All leaf hashes use BLAKE3, chosen for speed and cryptographic strength. Content is normalized before hashing (whitespace trimmed, consistent line endings) so that formatting-only changes don’t produce different hashes.

SPL dual hashing

SPL blocks receive two hashes:

content_hash — hash of the raw SPL text (changes on any edit, including comments)
ast_hash — hash of the parsed SPL AST (changes only on logical modifications)

The AST hash drives theory cache invalidation — see architecture/Cache. The content hash detects any change for drift purposes.

File and vault roots

Scanner

The scanner is zetl’s entry point for understanding a vault. It walks every Markdown file and standalone .spl file, extracting concepts/Wikilinks and concepts/Spindle Lisp blocks in a single pass.

(given wikilink-extraction)
(given spl-extraction)

Wikilink extraction

The scanner recognises all common wikilink forms:

[[Page]] — basic link
[[Page|alias]] — aliased link
[[Page#heading]] — heading anchor
[[Page^block-id]] — block reference
![[Page]] — embed

Extracted links feed into the Link Graph for query and traversal.

Exclusion zones

The scanner skips wikilink-like patterns that appear inside:

Fenced code blocks (```)
Inline code (`)
HTML comments ()
YAML frontmatter (--- fences)

This prevents false positives from code examples or metadata.

SPL extraction

Cache

Cache

zetl uses two-tier incremental caching for both the Link Graph and the Reasoning Engine’s theory.

(given mtime-based-cache)
(given incremental-rebuild)
(given disposable-cache)

How it works

The cache lives in .zetl/ at the vault root:

index.json — serialised link graph with per-file Merkle Tree roots
theory.json — serialised reasoning theory, conclusions, and grounding data

Two-tier invalidation

Tier 1 (mtime): If a file’s modification time hasn’t changed, skip it entirely — no I/O beyond the stat call
Tier 2 (content hash): If the mtime changed but the BLAKE3 content hash matches the cached value, skip reprocessing

This handles the common case where git checkout updates mtimes without changing content.

SPL-specific invalidation

For the theory cache, zetl uses the SPL AST hash (not content hash) to determine if re-reasoning is needed. This means:

Prose edits that don’t touch SPL → no theory rebuild
SPL reformatting (comments, whitespace) → no theory rebuild
SPL logical changes (new facts, modified rules) → theory rebuild

ADR-002 Search Without Index

ADR-002: Search Without Index

Status: Accepted

Context

zetl needs full-text content search across vault files (see Search Command). The question is whether to build an inverted index (like Lucene/tantivy) or scan files directly on each query.

Decision

Use index-free scanning — read and search files directly on each query.

Rationale

Simplicity — no index build step, no index format to maintain, no index invalidation logic beyond what architecture/Cache already handles
Correctness — results always reflect the current file contents; no stale index entries
Acceptable performance — scanning 10,000 files completes in under 2 seconds on commodity hardware (see Performance)
Consistency — zetl already has an mtime-based cache for the link graph; adding a second indexing system increases complexity disproportionately

Trade-offs

Slower than an inverted index for very large vaults (>50k files)
No ranking or relevance scoring — matches are returned in file order
Regex search performance depends on pattern complexity

Mitigation

The --path filter (from Agent Ergonomics) lets users restrict search to a subdirectory, which reduces scan scope. The --limit flag bounds result size.

Search Command

zetl search provides full-text content search across all Markdown files in the vault.

(given fulltext-search-done)
(given regex-support)

Usage

# Literal text search
zetl -d . search "defeasible"

# Regex search
zetl -d . search "reason(ing|ed)" --regex

# With surrounding context
zetl -d . search "Cache" --context 100

# Case-sensitive
zetl -d . search "SPL" --case-sensitive

# Restrict to a subdirectory
zetl -d . search "given" --path "theories/*"

Flags

Flag	Default	Description
`--context N`	0	Include N characters of surrounding text
`--limit N`	50	Max results to return
`--regex`	off	Interpret query as a regular expression
`--case-sensitive`	off	Require exact case match
`--all`	off	Search raw content (include frontmatter, code blocks)
`--path <glob>`	none	Restrict results to files matching glob

Content filtering

By default, search skips frontmatter and fenced code blocks to focus on prose. Use --all to search raw file content including concepts/Spindle Lisp blocks and YAML frontmatter.

Design decision

zetl uses index-free scanning rather than an inverted index. This trades query speed on very large vaults for zero build-time indexing cost and simpler architecture. See ADR-002 Search Without Index for the rationale.

Fuzzy page name matching

For finding pages by name rather than content, use Similar Command instead: