Building sciogen — a codebase intelligence layer for coding agents

TL;DR — sciogen indexes a codebase once into an embedded triple store, then answers questions like "who calls AuthService.login?" or "what breaks if I change this?" as sub-second lookups that return typed objects with file:line locations — not grep output, not raw source dumps. It never calls an LLM. It's to your codebase what Elasticsearch is to documents.

The problem: files are the wrong granularity

A coding agent understands a codebase the only way it can — by reading files. To learn the callers of one function, it opens dozens of files and pulls thousands of lines of source into its context window, most of it noise. Ask "what breaks if I change this signature?" and it does the same crawl again, from scratch, every single time.

That's backwards. The structure of a codebase — who calls what, what imports what, where a concept lives — doesn't change between questions. It should be computed once and then queried, the same way a database builds an index once and serves millions of lookups against it.

That's the whole idea behind sciogen.

What it feels like

You run one command inside your repo:

sciogen index .

Five stages run per file, and the whole graph lands in a .sciogen/ directory inside the repo. A single-file change re-indexes in under a second, because a SHA256 differ skips everything whose hash hasn't moved.

From then on, the questions are instant.

"What breaks if I change this?"

This is the query that used to mean reading forty files. Now it's one call:

sciogen impact AuthService.login

The center node is the symbol you touched. Impact radiates outward hop by hop — direct callers, then their callers — and the result is the union subgraph of everything that could break, each node carrying its exact file:line. Notice the edges aren't all the same: a call sciogen resolved with certainty is a solid line; one it couldn't statically prove (dynamic dispatch, an injected dependency) is dashed. Which brings me to the design decision I'm most happy with.

Confidence, not silence

Static analysis is incomplete — that's a fact, not a bug. A call to login() usually resolves cleanly to src/auth/service.py:AuthService.login. But some calls go through dynamic dispatch, monkey-patching, or dependency injection, and no static tool can prove where they land.

The tempting move is to drop those edges. That gives you a clean graph and a lying one. sciogen keeps them — with a low confidence score instead of full trust:

Edge kind	Confidence
IMPORTS, DEFINES, EXTENDS, direct CALLS	1.0
CALLS resolved via imports	0.8–1.0
CALLS via dynamic dispatch / injection	0.2–0.5

Filtering happens at query time, never at write time. The graph stores everything; the caller sets its own threshold. An agent that wants only rock-solid edges asks for min_confidence=0.8; one exploring blast radius takes them all and weighs them itself.

How it works, in one breath

sciogen index is a five-stage pipeline, and the ordering is the point:

1. tree-sitter parses the source — 100+ languages behind one API.
2. A normalizer converts each language-specific AST into a single universal IR. Python, JS, and Go all emit the same ClassNode / FunctionNode after this step — so every layer downstream is language-agnostic.
3. The symbol resolver runs after parsing, on purpose. Tree-sitter gives you syntax; it does not give you semantics. The resolver traces each reference to the exact definition it points to and scores its confidence.
4. The graph builder materializes typed nodes and edges into KuzuDB via Cypher.
5. A SHA256 differ backed by SQLite makes every re-run incremental.

Embeddings of normalized symbol summaries — name, params, docstring, call list, never the raw code — go into ChromaDB at four granularities (file, class, function, chunk), so broad intent queries and precise retrieval each hit the right resolution.

Everything is embedded. No servers, no ports, no daemon — just one .sciogen/ folder:

Store	Role
KuzuDB	Graph topology — nodes, typed edges, confidence scores
ChromaDB	Vector embeddings at four granularities
SQLite	File hashes, symbol table, schema version

Three modes of query, one deterministic engine

There's no ML routing inside sciogen deciding what you meant. The method you call is the mode:

# Structural — pure graph traversal on KuzuDB
graph.get_callers("AuthService.login")

# Semantic — cosine similarity on ChromaDB vectors
graph.search("password hashing logic")

# Hybrid — vector hits seed the search, the graph expands the neighborhood
graph.search("all code related to authentication", mode="hybrid")

Same from the terminal, for a quick human look:

sciogen search "password hashing"       # semantic
sciogen callers AuthService.login       # who calls this?
sciogen impact UserModel.find_by_email  # what breaks?
sciogen deps src/auth/service.py        # imports, transitive deps

The inversion that matters

Here's the part people get backwards: sciogen never calls an LLM. The LLM calls sciogen.

  the LLM  ⇄  agent host (Claude Code, Cursor…)  ⇄  sciogen  ⇄  .sciogen/ graph
  └ model ┘   └── owns the LLM connection ──────┘  └─ just a tool server ─┘

Your coding agent reaches the graph over MCP (the Model Context Protocol) — register it once and the agent gets typed tools: get_callers, analyze_impact, get_dependencies, search, get_diff_impact. One command wires it up so the agent queries the graph before editing and refreshes the index after:

sciogen setup-agent . --hooks

The token savings everyone talks about? sciogen is completely unaware of them. It has no concept of a token or a context window. The savings are a side effect of returning eight typed records instead of forty files. That happens entirely on the agent's side — which is exactly where it should.

And when you want to look yourself

Agents query the graph; humans want to see it. sciogen explore serializes the whole graph into a single self-contained HTML file and opens it in your browser — nodes colored by type, edges styled by confidence, sized by how many things reference them. It starts collapsed at the file level and expands on click, so a ten-thousand-symbol graph stays legible. No server, no port, consistent with everything else in the stack.

sciogen explore

Try it

pip install sciogen
cd your-project
sciogen index .     # build the graph (incremental after)
sciogen explore     # look at it
sciogen mcp .       # serve it to your agent

Add .sciogen/ to your .gitignore — it's machine-local and regenerable.

sciogen is MIT-licensed and on PyPI. Source and docs live at github.com/ayanbag/sciogen or ayanbag.github.io/sciogen/. If you build something on top of it, I'd love to hear about it.

---

Related Links

github.com

github/sciogen

The source code and documentation for sciogen, the codebase intelligence layer for coding agents.

ayanbag.github.io

Sciogen docs

The official documentation for sciogen, the codebase intelligence layer for coding agents.