Scores the knowledge graph that code-graph-rag (cgr) builds against ground truth, with no Memgraph required (an in-memory capturing ingestor drives GraphUpdater(...).run(force=True)).

Scores cgr's definition nodes and DEFINES/DEFINES_METHOD edges against a scope-aware Python ast oracle.

uv run python -m evals.cli --target codebase_rag

Writes evals/results/scores.csv and evals/results/diff.json. Node identity join is (kind, file, start_line).

Scores whether cgr attributes the right calls to the module (caller side). A call runs at module-load time -- and so belongs to the module -- iff it is a top-level statement, a decorator, or a default-argument expression, i.e. it is NOT inside a function body. The L3 execution trace cannot measure this: it records the innermost function frame as the caller and drops <module> frames, so module-level attribution is its structural blind spot. An ast oracle fills it.

uv run python -m evals.module_calls --target codebase_rag

How it works:

• Oracle (module_calls.oracle_module_calls): walks each file's AST modelling import-time execution. A call counts when it runs at module load: top-level statements, list/set/dict comprehensions (eager), decorators, argument defaults, and -- only when the file does not from __future__ import annotations -- argument/return annotations. It does NOT count function/method bodies, lambda bodies, or generator expressions (deferred until called or consumed). Class bodies stay at module scope. It collects the simple name of every such call whose callee is first-party (a name defined in the target), excluding dunders.
• cgr side (module_calls.cgr_module_calls): every CALLS edge whose caller is a Module node, keyed by (module_file, callee_simple_name); a constructor call resolved to a Class.__init__ method is credited to Class (a bare first-party function named __init__ is left as a filtered dunder).
• Score: precision/recall over (module_file, callee_simple_name) edges.

The exact-attribution guarantee is covered by test_eval_module_calls.py (precision == recall == 1.0 on a controlled fixture: a top-level call, a default-argument call, a __main__ call, and a nested call that must NOT be module-attributed).

On the whole codebase_rag target the metric is a lower bound that surfaces two real, separate cgr gaps (not attribution errors):

• Recall is bounded by constructor calls to first-party classes with no explicit __init__ (NamedTuple/dataclass/pydantic) -- cgr has no method node to point the call at, so no edge is emitted. Closing this needs constructor calls to target the class node (tracked with the dead-code Class work).
• Precision is bounded by the trie suffix-match fallback occasionally resolving a module-level call to an unrelated first-party name.

Measures whether cgr's static CALLS graph contains the call edges that actually fire at runtime.

uv run python -m evals.l3

How it works:

• Static side (cgr_graph.extract_cgr_calls): builds cgr's graph over the target package (default codebase_rag) and collects every CALLS edge.
• Traced side (calls_trace.trace_calls): runs cgr indexing a small fixture (evals/results/l3_workspace/fixture/, written by _write_fixture) under sys.settrace, recording every (caller, callee) where both are first-party functions in the target. This is a dynamic trace of cgr's own code executing — the fixture's only job is to drive cgr through diverse code paths.
• Recall = |traced ∩ static| / |traced|. missed = traced − static is written to evals/results/calls_diff.json. Two scopes are reported: all calls and explicit (excluding dunder callees).

Because the ground truth is an execution trace, recall is a sound lower bound: it can only credit cgr for call sites the fixture actually exercises. Enriching the fixture (more Python constructs, more languages) widens coverage and is the intended way to harden the metric.

When a function is wrapped by a functools.wraps decorator (e.g. cgr's @recursion_guard), calling it dispatches at runtime through the decorator's generic inner wrapper, so a naive trace records two edges:

caller            -> recursion_guard.decorator.wrapper      # the generic wrapper frame
recursion_guard.decorator.wrapper -> the_real_method        # wrapper calling func(...)

cgr's static graph instead "sees through" the decorator and records the single logical edge caller -> the_real_method, which is what a reader of the graph wants — the recycled wrapper is plumbing, not a meaningful call-graph node.

To keep the trace and the static graph in agreement, calls_trace._frame_qn attributes a wrapper frame to the function it wraps (recovered from the wrapper's closed-over callable, following any __wrapped__ chain). This turns caller -> wrapper into caller -> the_real_method and collapses wrapper -> the_real_method into a self-edge (which the tracer already drops). The decision is normalize in the eval, not model wrappers in cgr, so cgr's graph stays free of generic wrapper nodes.

Covered by codebase_rag/tests/test_l3_decorator_normalization.py.

Answers the question raised in issue #424: does graph-augmented retrieval find the code that calls a symbol better than plain grep? This is the retrieval layer decoupled from any LLM, which is the measurement the GitLab GKG evaluation (work item #224) flagged as out of scope. (That work item, contrary to a widely repeated claim, contains no "8% over grep" figure; its headline was an agentic SWE-bench-Lite pass rate of roughly 6 to 7 of 23 issues. This benchmark measures retrieval quality directly instead.)

uv run python -m evals.retrieval --target codebase_rag

The task: for every first-party symbol S, find the files that call S. The comparison unit is a file-level call edge (caller_file, callee_simple_name), which mirrors the GKG "did it open the right file" localization signal. Three conditions are scored against one Python ast oracle over the same file and first-party symbol universe:

• graph (retrieval.cgr_call_edges): every cgr CALLS/INSTANTIATES edge, reduced to its caller node's file and the callee's simple name (a constructor resolved to Class.__init__ is credited to Class, as in L2).
• grep_name (retrieval.grep_call_edges, GrepMode.NAME): ripgrep for the bare symbol token \b(name)\b, the first thing a user reaches for.
• grep_call (GrepMode.CALL): ripgrep for the symbol followed by a paren \b(name)\s*\(, a call-tuned pattern.
• Oracle (retrieval.oracle_call_edges): every ast.Call whose callee simple name is first-party and non-dunder, attributed to its file, plus every bare read of a first-party @property/@cached_property getter (first_party_property_names) — such an access invokes the getter, which cgr models as a CALLS edge, so the oracle counts it too rather than scoring cgr's property-as-call edges as false positives.

Requires rg (ripgrep) on PATH; evals.retrieval exits cleanly if it is missing. Writes evals/results/retrieval_scores.csv and evals/results/retrieval_diff.json. The thesis and grep's two failure modes (a bare reference or import counts as a hit, and a definition site def S( is indistinguishable from a call) are pinned by codebase_rag/tests/test_retrieval_eval.py.

Both grep conditions are name-based, so any called name is present textually. Graph recall below 1.0 reflects the few call edges cgr does not resolve; graph false positives are call edges cgr emits that the oracle's notion of a call does not see.

On a large external corpus (django/django, ~2900 files) the resolved graph scores precision 0.977, recall 0.938, F1 0.957, decisively ahead of grep (grep_call F1 0.789, grep_name F1 0.506). The property-aware oracle is what makes that precision faithful: before it, cgr's correct property-getter edges (e.g. self.related_model on a @cached_property) were scored as ~2000 false positives, understating precision to 0.838. cgr's property-as-call is receiver-aware (a bare method reference or a plain attribute that merely name-collides with another class's property emits nothing). The residual false positives are a diffuse tail (property setter decorators, gettext aliasing, base-class edges); the recall tail is first-party names colliding with builtins (django's own def super(self) makes every builtin super() call look like a missed first-party edge), the same name-based limitation documented for the per-language retrieval evals below.

Answers a correctness question the other layers cannot: after cgr re-indexes only the files that changed, does the resulting graph still equal a clean full re-index of the same tree? Incremental indexing is where a knowledge graph silently rots, so the clean re-index is the oracle and any divergence is a real bug.

uv run python -m evals.incremental --target codebase_rag --sample 25

The probe is a semantically neutral edit: a trailing comment is appended to one file, changing its hash (so cgr treats it as modified) without changing its AST (so a clean re-index of the edited tree is identical to the original). For each sampled file the harness indexes a fresh copy, applies the neutral edit, runs an incremental update, then compares the mutated graph node for node and edge for edge against a clean forced re-index of the identical on-disk state.

The comparison runs against a faithful in-memory store (cgr_graph._StatefulIngestor) that implements the exact delete and fetch Cypher the incremental updater issues (DETACH DELETE of a changed file's Module subtree, file and folder deletes, orphan-external pruning, and the prune path queries), so deletions take real effect rather than being mocked away. The store's semantics are pinned by codebase_rag/tests/test_incremental_eval.py; the same suite also pins the runner's requirement to purge any pre-existing hash cache copied from the source tree, without which the baseline index would skip every file.

What it surfaced and drove a fix for: issue #532. Editing a file DETACH-deletes its Module subtree, including the reference edges incident on its functions. The eval showed the loss was broader than the issue recorded: inbound CALLS/IMPORTS/INSTANTIATES from unchanged callers were deleted and never rebuilt (the callers are not reprocessed), and a fresh incremental run also rebuilt the function registry from changed files only, so even the changed file's outbound calls to symbols defined in unchanged files were dropped. The fix, verified by this eval, has two parts:

• Inbound edges are captured before deletion and restored verbatim (rather than re-resolved, which would diverge: cgr resolution is context-sensitive).
• Outbound resolution rehydrates the function registry from the persisted graph so calls into unchanged files resolve again.
• Property and inheritance context is rehydrated too: the @property status of methods and the class_inheritance hierarchy (from persisted INHERITS edges) are restored before call resolution, so a re-parsed caller still resolves property access and protocol dispatch into unchanged files. Each INHERITS edge persists a base_index, and rehydration orders the bases by it, so multiple inheritance keeps its original source order (method resolution and override attribution are first-match-wins over the base list, so a lost order would bind a call to the wrong base).

Residual divergence is now confined to a small tail of the changed file's own calls resolved through deeper type inference, and to import-granularity differences; these are documented as follow-ons, not regressions. Writes evals/results/incremental_scores.csv and evals/results/incremental_diff.json.

Inbound capture is intentionally scoped to re-indexed files (changed, not new or deleted), because a re-indexed file keeps its module qualified name, so the restore target still exists after reprocessing. Moved or renamed files are not captured by design: the old path is deleted and the new path is new, so an unchanged caller's import of the old name no longer resolves, exactly as in a clean re-index, and dropping that now-dangling edge is correct. Restoring edges for a vanished module qn would instead fabricate a phantom module node, so the scoping is the safe choice rather than a gap. A transparently re-exported rename (old name still resolves) is the one narrow case left to a clean re-index.

The structural L1 above grades internal IMPORTS edges by their resolved target file. It does not check how cgr classifies the other imports: stdlib and third-party. This eval does, against an ast plus filesystem oracle, to surface internal/external misclassification (the shape of issue #498).

uv run python -m evals.import_resolution --target codebase_rag

The comparison unit is (importing_file, top_level_package, is_external). Both sides reduce an import to its top-level package name the same way (import numpy.linalg and from numpy import x both reduce to numpy), and both decide internal versus external by whether that top level is the project package, so the oracle is independent of cgr's own resolver. cgr models an external import as a Module node flagged is_external=True linked by IMPORTS (it does not emit ExternalPackage/DEPENDS_ON_EXTERNAL for code-level imports), so the eval reads the flag off each IMPORTS target. from __future__ import ... is a compiler directive rather than a dependency and is excluded on both sides (a calibration the tests pin). Writes evals/results/imports_scores.csv and evals/results/imports_diff.json; the oracle and the misclassification signal are pinned by codebase_rag/tests/test_import_resolution_eval.py.

The structural L1 grades INHERITS by the base's simple name. This eval grades two deeper things against an ast oracle: that cgr resolves a base to the correct first-party class qualified name (INHERITS target), and that method overrides are attributed to the right base class (OVERRIDES).

uv run python -m evals.inheritance --target codebase_rag

The oracle resolves a base only when it is unambiguous: defined in the same module or imported via from <first-party> import <Base>. Attribute bases (pkg.Base), star-imported, and external bases are skipped and counted, never guessed. Two deliberate scope limits keep the oracle honest rather than noisy:

• INHERITS is graded only for top-level classes (the universe the oracle enumerates); cgr edges whose subclass is a class nested inside a function are not graded against an oracle that never saw them.
• OVERRIDES is graded only for single-inheritance classes, where "which base does method m override" is unambiguous. Multi-base mixin classes are excluded on both sides, because the answer there is decided by the MRO, which this ast oracle does not model.

Writes evals/results/inheritance_scores.csv and evals/results/inheritance_diff.json; pinned by codebase_rag/tests/test_inheritance_eval.py.

The retrieval eval folds class instantiation into its CALLS localization (a constructor call resolves to Class.__init__, credited to the class). This eval grades cgr's INSTANTIATES edges on their own, so a constructor-resolution regression is visible separately from ordinary calls.

uv run python -m evals.instantiation --target codebase_rag

The unit is (caller_file, class_simple_name). The oracle counts every ast.Call whose callee simple name is a first-party class, excluding bare-name calls whose name is rebound in that file by a non-first-party import (from ext import Config; Config() names the external Config, not a same-named first-party class). cgr contributes its INSTANTIATES edges reduced to the caller's file and the class simple name. Writes evals/results/instantiation_scores.csv and evals/results/instantiation_diff.json; pinned by codebase_rag/tests/test_instantiation_eval.py.

Making the oracle import-aware surfaced a cgr precision bug: a constructor whose name was explicitly imported from an external module (from evals.types_defs import GraphData, with evals outside the indexed project) was resolved by the simple-name trie fallback to a same-named first-party class (codebase_rag.types_defs.GraphData), emitting a wrong INSTANTIATES edge. Fixed in call_resolver.py (_is_external_import suppresses the trie fallback for a bare name bound to a genuinely external import; first-party imports, prefixed or bare, are unaffected). On codebase_rag: precision rose from 0.976 to 1.000 (9 false edges removed), recall 0.997. The one remaining miss is a class defined in a test method and instantiated from inside a nested class's method (a closure over an enclosing-function scope), a known resolution gap left documented rather than scoped away.

cgr's dead-code command reports functions/methods unreachable from any entry point. It runs as a Cypher reachability query against the database, which the deterministic in-memory harness cannot execute, so this eval faithfully re-implements that query's reachability over the captured graph and grades it on controlled fixtures whose dead set is known by construction.

uv run python -m evals.dead_code --target codebase_rag   # informational report

Roots are project functions that are decorated with an entry-point decorator (@app.route and friends), exported, named as an entry point, reached by a Module via CALLS (a module-level call), or in a test file when tests are included; everything reachable from a root via CALLS (plus INSTANTIATES / INHERITS with classes) is live, and the rest is dead. The reachability is unit-tested on hand-built graphs, so when it is run over a cgr-built graph from a fixture with a known dead set, a mismatch indicts cgr's CALLS graph (a missing edge would flag a live function as dead) rather than the scorer. The graded eval is the fixture suite codebase_rag/tests/test_dead_code_eval.py; the CLI's corpus mode is informational only, because a real repo has no independent dead-code oracle (true reachability needs the very call graph under test). On codebase_rag it currently reports 4450 unreachable functions/methods (tests excluded).

Every other eval runs on a single top-level package (codebase_rag), so none checks the case cgr is built for: a monorepo with several top-level packages where one references another. This eval extracts cgr's CALLS and IMPORTS edges whose endpoints live in different top-level packages and grades them on synthetic multi-package fixtures whose cross-package edges are known by construction (codebase_rag/tests/test_cross_project_eval.py). It confirms that pkg_b.use.run() calling pkg_a.core.shared(), and pkg_b/pkg_c importing pkg_a modules, resolve across the package boundary, while intra-package edges are correctly excluded. cgr resolves all of these; the eval stands as a regression guard for monorepo cross-package resolution.

Every other eval indexes a single language (or a Python-dominant tree), so none checks what happens when cgr builds one graph over files from all 14 supported languages at once — the mixed-language repo it is actually pointed at in the wild. This eval ingests a corpus spanning every SupportedLanguage, with a deliberate three-way basename collision (shapes.rs / shapes.cpp / shapes.ts), and grades cross-language integrity invariants that need no external oracle (codebase_rag/tests/test_polyglot_eval.py):

1. every available language contributes at least one module and one definition (no language silently dropped when mixed in),
2. files that strip to the same module qn get distinct qns — the collision is disambiguated, not overwritten under the qualified_name constraint (issue #652 class),
3. no CALLS / INHERITS / OVERRIDES / IMPLEMENTS / INSTANTIATES edge crosses a language boundary (a Rust call resolving onto a Python node would be cross-language qn bleed),
4. the recorded graph is dangling/orphan free (the create_and_run_updater audit), and the report is deterministic so the collision winner never churns.

cgr satisfies all of these on a clean (full) build; the eval stands as a regression guard for polyglot ingestion. The collision winner on a full build is decided by sorted file order (shapes.cpp keeps the bare …shapes, shapes.rs and shapes.ts take the suffixed form).

Grades cgr's CALLS graph at function granularity against an ast oracle that resolves only the calls a reader can resolve without type inference: a bare-name call (foo()) whose target is a first-party function reached via a from ... import foo or a same-module top-level def. Each becomes a (caller_qn, callee_qn) edge. Method / attribute / dynamic calls need cgr's type inference and are out of the oracle's scope, so only recall is graded (cgr resolving more than static analysis can is expected, not a false positive). The oracle uses ast import resolution, not cgr's function-registry trie, so it is independent.

uv run python -m evals.static_calls --target codebase_rag

Decorator applications (@deco(...)) are excluded (they are not calls the decorated function makes), and the oracle attributes a call to its real enclosing function qn including nested scopes (Class.method.nested).

This eval caught a real cgr bug. A call inside a function nested in a method was emitted with a caller qn that dropped the method (Class.nested instead of Class.method.nested, matching no node), and was also over-attributed to the enclosing method. After the root-cause fix in call_processor.py (_class_member_qn_and_label + _calls_owned_by), recall on codebase_rag is 4434/4434 = 1.0. Pinned by codebase_rag/tests/test_static_calls_eval.py and the regression codebase_rag/tests/test_nested_method_call_qn.py.

The retrieval benchmark above is Python-only. This extends file-level call localization to a second language: for each first-party Go symbol, which files call it. cgr's Go CALLS edges, reduced to (caller_file, callee_simple_name), are graded against call sites extracted by Go's own go/ast (the same oracle program as the Go L1 structure eval, extended to emit CallExpr callees), over the same first-party name universe. The oracle uses Go's standard parser, fully independent of cgr's tree-sitter Go frontend, so this measures cgr's cross-file Go call resolution against ground truth.

uv run python -m evals.go_retrieval --target <go-sources>

Requires the go toolchain on PATH; the eval exits cleanly if it is missing. The oracle counts a call by its callee simple name (a bare foo() or the selector tail of x.Method() / pkg.Func()), keeping only callees that are declared first-party functions/methods, exactly mirroring the Python retrieval oracle. Pinned by codebase_rag/tests/test_go_retrieval_eval.py, where cgr's Go call graph matches the go/ast oracle on the fixture. The same harness shape generalizes to the other native-oracle languages (Rust, TypeScript, Java) by teaching each oracle to emit call sites.

Running this on a real stdlib package (encoding/json via GOROOT) instead of the fixture first surfaced two Go call-graph bugs, then drove their fix to a clean result. (1) Receiver methods got a receiver-dropping caller qn that bound to no node; fixed by _go_receiver_method_caller. (2) Go receiver dispatch (d.method(), a method call on a receiver, parameter, or composite-literal local) resolved to no callee at all, because Go calls were not typed and the Go selector_expression callee node was never read by _get_call_target_name. Fixed by adding Go to the typed-language set, a Go local-variable type inference engine (parsers/go/type_inference.py) that maps receivers/parameters/:= locals to their type, and reading the selector callee name. On encoding/json, precision/recall went from 1.0/0.55 to 1.0/1.0 (110/110 call edges, zero false positives).

The same harness applied to Rust: for each first-party Rust symbol, which files call it. cgr's Rust CALLS edges, reduced to (caller_file, callee_simple_name), are graded against call sites extracted by syn (the de-facto Rust parser, the same oracle as the Rust L1 structure eval, extended to emit ExprCall path callees and ExprMethodCall method idents), over the same first-party name universe. syn is independent of cgr's tree-sitter Rust frontend.

uv run python -m evals.rust_retrieval --target <rust-sources>

Requires the cargo toolchain on PATH; the eval exits cleanly if it is missing. Pinned by codebase_rag/tests/test_rust_retrieval_eval.py, where cgr's Rust call graph matches the syn oracle on the fixture.

Running it on a real stdlib module (core::str, via the rust-src component under the rustup sysroot) surfaced two cgr bugs and drove the fix from precision/recall 0.91/0.65 to 0.94/0.95. (1) A method in a generic impl block (impl<'a> Thing for Chars<'a>) was attributed to a caller qn that carried the impl's generics (crate.lib.Chars<'a>.go), but the method node is registered on the bare type (crate.lib.Chars.go), so every such CALLS edge had a dangling caller and was dropped; fixed by routing _get_rust_impl_class_name through the same rs_utils.extract_impl_target the definition pass uses (recovered 44 of 58 missing edges). (2) A regression from the externally-imported-name fix: _is_external_import mistook Rust relative imports (use super::b::helper, whose recorded target is the ::-separated super::b::helper) for external symbols and suppressed the trie fallback, dropping the call; fixed by restricting that guard to dotted absolute-path imports (Python/Java form), leaving ::-path and relative imports to the trie. Pinned by codebase_rag/tests/test_rust_impl_method_call_qn.py. The remaining gap is field/generic method dispatch (self.field.method(), Pattern trait calls) needing deeper Rust type inference, plus oracle undercount inside macro bodies (write! expands to .fmt() calls syn does not see), documented rather than scoped away.

The same harness applied to Java: for each first-party Java symbol, which files call it. cgr's Java CALLS edges, reduced to (caller_file, callee_simple_name), are graded against method-invocation sites extracted by javac (the JDK Compiler Tree API, the same oracle as the Java L1 structure eval, extended to emit the trailing identifier of each MethodInvocationTree), over the same first-party name universe. javac is independent of cgr's tree-sitter Java frontend.

uv run python -m evals.java_retrieval --target <java-sources>

Requires the javac/java toolchain on PATH; the eval exits cleanly if it is missing. Pinned by codebase_rag/tests/test_java_retrieval_eval.py, where cgr's Java call graph matches the javac oracle on the fixture.

Running it on a real stdlib package (java.util, 349 files from the JDK src.zip) surfaced three cgr bugs and drove recall from 0 (every Java call dropped) to 0.52 at precision 1.0 (zero false positives). (1) The definition pass registers a Java method node with its parameter signature (Class.name(args) — Java overloads), but the call pass built the enclosing-method caller qn without it (Class.name), so every Java method's CALLS from-endpoint matched no node and the edge would not attach in Memgraph; fixed by routing the caller qn through the same signature build the definition pass uses (codebase_rag/tests/test_java_call_caller_qn.py). (2) find_package_start_index returns None for any project not under a recognized src/main/java layout, so _build_fqn_lookup_map left the simple-name to module map empty and all cross-file resolution (instance dispatch in sibling files) silently failed; fixed by falling back to the segment after the project root for flat/non-standard layouts. (3) A static call on a bare class-name receiver in a sibling file (T.make(), same package, no import) never resolved because the receiver-type lookup only checked the current module and explicit imports; fixed with a same-package class-name fallback in _resolve_java_object_type. The remaining gap is interface/abstract dispatch (the name-based oracle counts a call whenever the callee name is any declared first-party method, but cgr emits an edge only when the concrete receiver type is statically knowable) and deep receiver-type inference (iterator/functional-interface/generic element types), documented rather than scoped away.

The same harness applied to TypeScript: for each first-party TS symbol, which files call it. cgr's TS CALLS edges, reduced to (caller_file, callee_simple_name), are graded against call sites extracted by the TypeScript compiler API (the same oracle as the TS L1 structure eval, extended to emit the callee identifier of each CallExpression — bare identifier or property-access trailing name), over the same first-party name universe. tsc is independent of cgr's tree-sitter TS frontend. The oracle also now names arrow / function expressions by their binding (const foo = () => … → foo), matching how cgr names them, so they enter the declared-name universe.

uv run python -m evals.ts_retrieval --target <ts-sources>

Requires node/npm (the typescript dependency installs on first run); the eval exits cleanly if node is missing. Pinned by codebase_rag/tests/test_ts_retrieval_eval.py, where cgr's TS call graph matches the tsc oracle on the fixture.

Running it on a real library (zod, 88 non-test files from packages/zod/src/v4) surfaced two cgr bugs and drove recall from 0.45 to 0.75 at precision 1.0 (zero false positives). (1) Exported-function duplication: the definition pass already ingests an exported function / const-arrow at its natural qn, but the ES6-export pass (ingest_exported_function) re-registered it, minting a spurious qn@line duplicate node (register_unique_qn collision) onto which call resolution then bound — so callers of the natural node were invisible and ~half of all TS call edges carried a mangled name@line callee. In an ESM codebase where most functions are exported this polluted the whole graph; fixed by skipping the export-pass registration when the natural qn already exists (codebase_rag/tests/test_ts_export_no_duplicate.py). (2) Arrow-body calls unresolved: the call pass derived a caller name with child_by_field_name("name"), which is empty for an arrow / function expression, so it skipped the whole const f = () => … body and never emitted its calls — and modern TS is mostly arrow functions. Fixed by recovering the binding name for the variable_declarator form so the body's calls attribute to the same qn the definition pass registered (codebase_rag/tests/test_ts_arrow_caller_calls.py). A follow-up fix also models class-field arrow members (class T { helper = () => … }): the definition pass dropped them (the arrow has no name field, so ingest_method returned early) so no T.helper node existed and its body's calls were lost. The binding name is now taken from the enclosing field definition in both the definition and call passes, modelling it as T.helper like a normal method (codebase_rag/tests/test_ts_class_field_arrow.py). The remaining gap is calls inside anonymous (returned/inline) arrows, method dispatch on typed receivers, and the name-based oracle's over-count of common method names (error on a receiver cgr cannot type) — documented rather than scoped away.

The same harness over .js/.jsx: the tsc oracle parses JavaScript syntactically (independent of cgr's tree-sitter JS frontend), so it grades cgr's JS CALLS resolution against ground truth for a language cgr handles through a partly distinct path (CommonJS require/module.exports, prototype methods, no type annotations).

uv run python -m evals.js_retrieval --target <js-sources>

Requires node/npm; pinned by codebase_rag/tests/test_js_retrieval_eval.py (cgr matches the oracle on a clean ES-module fixture).

Running it on a real corpus (webpack/lib, 577 files, classic CommonJS) surfaced one dominant cgr bug and drove recall from 0.61 to 0.83 at precision ~1.0. Calls inside an anonymous arrow passed as an argument were dropped: a callback like compiler.hooks.compilation.tap(NAME, (compilation) => { getReplacements() }) has no name and no binding, so the call pass skipped the arrow, and _calls_owned_by had already excluded its calls from the enclosing method — so the call attributed to nothing and vanished. Since webpack (and most modern JS) is saturated with this hooks.tap(name, (x) => …) pattern, it was the leading recall killer. Fixed by excluding only attributable nested functions (named, or arrows bound to a name) from the enclosing scope, so an anonymous arrow's calls bubble up to the nearest named function/method — matching where the oracle attributes them (codebase_rag/tests/test_js_anonymous_callback_calls.py). The remaining gap is method dispatch on receivers cgr cannot type and the name-based oracle's over-count of calls through function parameters (callback/fn) and Node built-ins (join/resolve) whose names collide with first-party declarations — documented rather than scoped away.

The same harness applied to PHP: for each first-party PHP symbol, which files call it. cgr's PHP CALLS edges, reduced to (caller_file, callee_simple_name), are graded against call sites extracted by php-parser (the same oracle as the PHP L1 structure eval, extended to emit the callee of each call — a bare function name, or the trailing member of a $obj->m() / Class::m() lookup — and to carry real declaration names), over the same first-party name universe. php-parser is a pure-JS PHP parser, independent of cgr's tree-sitter PHP frontend.

uv run python -m evals.php_retrieval --target <php-sources>

Requires node/npm (the php-parser dependency installs on first run; no PHP runtime needed). Pinned by codebase_rag/tests/test_php_retrieval_eval.py, where cgr's PHP call graph matches the php-parser oracle on the fixture.

Running it on a real library (monolog, 121 files from src/Monolog) surfaced one cgr bug and drove recall from 0.97 to 0.99 at precision 1.0 (zero false positives). Plain function calls were never resolved: a PHP function_call_expression carries its callee as a name node under the function field, a type _get_call_target_name did not handle, so no callee name was extracted and the edge was dropped — only method ($obj->m()) and static (Class::m()) calls, which expose a name field directly, resolved. cgr's strong OOP-PHP score had masked the gap (monolog is almost entirely methods). Fixed by adding the PHP name type to the callee match arm (codebase_rag/tests/test_php_function_call.py). The remaining gap is the name-based oracle's over-count: monolog declares first-party methods named substr/reset/fwrite (e.g. a UTF-8-aware Utils::substr), so a bare call to the PHP builtin of the same name is counted as first-party — cgr's simple-name trie fallback links it to the same-named first-party symbol, which the file+name metric credits. Documented rather than scoped away.

The same harness applied to Lua: for each first-party Lua function, which files call it. cgr's Lua CALLS edges, reduced to (caller_file, callee_simple_name), are graded against call sites extracted by luaparse (the same oracle as the Lua L1 structure eval, extended to emit the callee of each call — a bare name, or the trailing member of a t.f() / t:m() lookup — and to carry real declaration names, including function expressions named after their assignment target), over the same first-party name universe. luaparse is a pure-JS Lua parser, independent of cgr's tree-sitter Lua frontend.

uv run python -m evals.lua_retrieval --target <lua-sources>

Requires node/npm (the luaparse dependency installs on first run; no Lua runtime needed). Pinned by codebase_rag/tests/test_lua_retrieval_eval.py, where cgr's Lua call graph matches the luaparse oracle on the fixture.

Running it on a real library (penlight, 39 files from lua/pl) surfaced one cgr bug and held precision at 1.0 (zero false positives), recall 0.93. Calls inside function expressions were never attributed: a function bound to a variable or table field (M.runner = function() ... end) has no name field, so the call pass could not derive the caller qn and skipped the whole body — the same family as the JS/TS arrow-caller gap. The definition pass already names these after their assignment target (lua_utils.extract_assigned_name); the fix reuses that in the call pass so the body's calls attribute to the right node (test_cgr_resolves_function_expression_body_calls). The remaining recall gap is colon-method dispatch and the name-based oracle's over-count: a colon method function List:append(i) gets cgr qn List:append (simple name List:append, not append), and a call obj:append(x) on a receiver whose type cgr cannot statically infer (e.g. res[klass]:append(val)) cannot resolve, while the oracle credits any call whose simple name is a declared first-party function. Documented rather than scoped away.

The same harness applied to C: for each first-party C function, which files call it. cgr's C CALLS edges, reduced to (caller_file, callee_simple_name), are graded against call sites extracted by libclang, over the same first-party name universe. cgr parses C with tree-sitter by default (CPP_FRONTEND=libclang is off), so libclang — a full C front end — is an independent oracle. C has no overloading, so a simple name is unambiguous.

uv run python -m evals.c_retrieval --target <c-sources>

Requires libclang (the clang.cindex Python bindings). No compile_commands.json is needed: each .c file is parsed directly, with the SDK sysroot (xcrun --show-sdk-path) and clang's builtin-header directory (clang -print-resource-dir) added so system and compiler headers resolve. A translation unit that still emits an error diagnostic (a missing build-generated header such as a configured config.h) has a possibly-truncated AST, so the oracle abstains on it: that file is left out of the covered set and the cgr side is held to the same files (the count of graded files is logged, never silently dropped). Pinned by codebase_rag/tests/test_c_retrieval_eval.py, where cgr's C call graph matches the libclang oracle on a header-free fixture.

Running it on a real project (jq, 18 of 21 src/*.c files parsed cleanly; the other three need build-generated headers) gives precision 0.98, recall 0.93, F1 0.96. Every diff entry was classified against the source; the tail splits into two causes.

Most of it is the C preprocessor gap (inherent — cgr's tree-sitter front end does not evaluate #if/macros, while libclang does), confirmed in both directions:

• All 11 false positives are calls cgr emits inside inactive conditional branches that libclang correctly compiles out: jvp_strtod / tsd_dtoa_context_get in jv.c under #ifdef USE_DECNUM, jv_parser_new in jq_test.c under #ifdef HAVE_PTHREAD, jv_mem_calloc in jv_print.c under #ifdef WIN32, yysymbol_name in parser.c under #if YYDEBUG (YYDEBUG is 0). cgr makes zero misresolutions — each is correct code that is simply dead.
• Most false negatives are calls that exist only after macro expansion, which cgr cannot see: the jv_object_foreach macro (jv.h) expanding to jv_object_iter* calls, the token() / check_done() macros in jv_parse.c, flex's input macros, and bison's api.prefix {jq_yy} rename (libclang records jq_yylex after the #define; cgr resolves the same call under the textual name yylex).

The rest was a tree-sitter-c grammar bug, now fixed in cgr via patched forks: in the bison-generated parser.c, tree-sitter produced 20 ERROR nodes, and the normal yyerror definition (a plain void yyerror(...) at parser.c:406) fell inside one, so cgr never registered yyerror as a function and every call to it was unresolved. The trigger was reduced to a block comment between two tokens of a #define value: preproc_arg is a single token whose regex stops before /*, but the grammar allowed only one value token, so the text after the comment had no rule to attach to. Tracked in issue #555 and upstream as tree-sitter-c #319 (open, unreviewed). Because upstream review is slow, cgr sources its grammars from patched forks (tree-sitter-c and tree-sitter-cpp, pinned via [tool.uv.sources]) that change the value to repeat($.preproc_arg) so comment extras can sit between segments. With the patched grammar, yyerror is captured (parser.c ERROR count drops from 20 to 12; the remainder are attribute-macro constructs that only preprocessing can resolve). Drop the overrides once the fix lands upstream.

The same harness applied to C++: for each first-party C++ function or member function, which files call it. cgr's C++ CALLS edges, reduced to (caller_file, callee_simple_name), are graded against call sites extracted by libclang, over the same first-party name universe (free functions, function templates, and member functions; constructors/destructors are excluded because cgr models object creation as INSTANTIATES). cgr parses C++ with tree-sitter by default (CPP_FRONTEND=libclang is off), so libclang is an independent oracle. Overloads collapse under the (file, simple-name) metric, so they need no disambiguation.

uv run python -m evals.cpp_retrieval --target <cpp-sources> --define LEVELDB_PLATFORM_POSIX=1

Requires libclang. C++ standard headers must be parsed by a libclang whose clang version matches the active SDK's libc++; the bundled pip wheel's older clang cannot, so the oracle prefers a system libclang (/Library/Developer/CommandLineTools/usr/lib/libclang.dylib on macOS) and pins it before the first parse. No compile_commands.json is needed: each source is parsed directly with the SDK sysroot, the SDK's libc++ headers (which must precede clang's builtin resource headers), and every first-party header directory added as an include path. A build normally supplies platform macros (e.g. LEVELDB_PLATFORM_POSIX); pass them with --define. A translation unit that still emits an error diagnostic abstains (left out of the covered set; the cgr side is held to the same files, the graded count logged). To avoid crediting or penalizing calls whose simple name merely collides with a first-party symbol, the oracle grades a call only when libclang resolves its callee to a first-party declaration (child.referenced), so a std::string::size() call is never counted as a first-party size edge. Pinned by codebase_rag/tests/test_cpp_retrieval_eval.py, where cgr's C++ call graph matches the libclang oracle on a header-free namespaced fixture.

Running it on a real project (leveldb, 40 of 42 core sources parsed cleanly; the other two are Windows-only or need gmock) gives precision 0.99, recall 0.83, F1 0.90 — recall up from 0.54 before the namespace fix below, and precision lifted from 0.96 by the receiver type inference chain (next section).

The dominant gap was a real cgr bug: the call pass dropped the namespace from the caller qn. The definition pass binds a C++ free function or class inside a namespace to a namespaced qualified name (module.ns.fn, module.ns.Class), but the call pass built the enclosing caller's qn without the namespace (module.fn, module.Class.method). Every such CALLS edge's source therefore pointed at a node that does not exist (904 of 1227 C++ call sources dangled on leveldb, all of it in namespace leveldb), so the call never attached. The fix routes both the free-function and class qns through the same cpp_utils.build_qualified_name the definition pass uses, so caller and node qns always agree (RED test test_cpp_namespace_call_caller_qn.py). Dangling sources fell to 251 and recall rose 0.54 → 0.82.

Three follow-on cgr fixes then landed on top of the namespace fix, building out C++ receiver type inference: (a) C++ joined the typed-language set with an engine (parsers/cpp/type_inference.py) mapping parameters and local variables — including reference declarators, multi-variable declarations, and drop-on-conflict lexical scope handling — to their class types, so obj->method() resolves to the method on the receiver's class instead of by bare name; (b) a member call whose receiver has a known external type (a std::string) skips the name-only trie fallback instead of mis-binding to a same-named first-party method (call_resolver._receiver_type_is_external); (c) member fields are captured per class at ingestion (build_field_type_map, stored in class_field_types keyed by class qn) and merged into the receiver map by the caller's enclosing class, so mutex_.Lock() in an out-of-line method resolves to the field's type even though the field is declared in a different file (the header). Together these lifted precision 0.96 → 0.99 (false positives 30 → 10) on leveldb.

The remaining tail is documented, not scoped away:

• Operator overloads (operator= ×25, operator[], operator==/!=, 23% of the misses): libclang records a = b and a[i] as calls to the overloaded operator methods, while cgr models them as builtin.cpp.* operator calls — a metric difference, not a misresolution.

• tree-sitter-cpp parse corruption in complex files (the dominant residual FPs). Traced to a tree-sitter-cpp grammar limitation, not a cgr resolution bug. A preprocessor conditional inside a construct — the reduced minimal trigger is a constructor member-initializer list interrupted by #if/#endif:

  class A { public: A(int n) :
  #if !defined(NDEBUG)
      x_(n),
  #endif
      y_(n) {} int x_; int y_; };

  emits ERROR nodes (4 vs 0 without the #if). In real files this cascades: util/env_posix.cc (24 ERROR nodes) mis-nests every sibling class under the first (Limiter.PosixEnv), turns struct ::flock references into phantom classes, and degrades out-of-line methods like DBImpl::BuildBatchGroup into free functions that lose member-field inference — which is what produces the residual begin/end and Schedule/SetReadOnly* false positives. This exact trigger is already reported upstream as tree-sitter-cpp #297 ("ifdef not supported inside constructor member initializer list"); it is a sibling of the tree-sitter-c grammar bug tracked in #555. It is an emergent cascade — every isolated small reproduction (clean out-of-line methods, same-basename .h/.cc, an unrelated #if error in the same file) resolves correctly, so there is no minimally reproducible cgr-side fix; the fix is upstream (grammar) or the libclang frontend.

• The libclang frontend trades this precision gain for recall. Running the same leveldb benchmark with CPP_FRONTEND=libclang (a compile_commands.json from CMake) parses the corrupt files correctly — env_posix classes nest properly and the FP count drops to 1 (precision 0.9975) — but recall falls to 0.46 (F1 0.63): the frontend emits far fewer CALLS edges than the tree-sitter resolver, a diffuse gap (implicit operators, compile-DB parse-context differences from the oracle's per-file parse, and header-inline caller-file attribution). So the default tree-sitter path remains the higher-F1 choice; the libclang frontend is the correct frontend for parse fidelity but needs its own call-edge recall work before it is a drop-in improvement.

The same harness applied to Scala: for each first-party Scala def, which files call it. cgr's Scala CALLS edges, reduced to (caller_file, callee_simple_name), are graded against call sites extracted by scalameta, over the same first-party name universe (every declared def). cgr parses Scala with tree-sitter, so scalameta (the Scala compiler front-end's own parser, run via scala-cli) is an independent oracle. The oracle grades only files it parses cleanly (covered set; the cgr side is held to the same files), so a file using Scala 3 syntax that the 2.13 parser rejects penalizes neither side.

uv run python -m evals.scala_retrieval --target <scala-sources>

Requires scala-cli on PATH (it fetches scalameta on first run); the eval exits cleanly if it is missing. Pinned by codebase_rag/tests/test_scala_retrieval_eval.py, where cgr's Scala call graph matches the scalameta oracle on the fixture.

It surfaced a real cgr bug: infix-operator calls were dropped. cgr's Scala spec lists infix_expression as a call node type, so it is collected, but the callee-name extractor (call_processor._get_call_target_name) had no case for it — an infix_expression (a ~> b, xs map f) exposes its callee through an operator field, not the function field the extractor keyed on, so every infix operator call (pervasive in a combinator DSL) returned no name and never attached. The fix adds a Scala-gated operator-field case. On scala-parser-combinators recall rose 0.40 → 0.73, F1 0.57 → 0.84, precision held at 1.0; scopt shows the same shape (precision 1.0, recall 0.71).

Scope: application and infix call sites, not bare paren-less selects. Scala's uniform access makes a nullary method call (obj.done) and a plain field read (obj.done where done is a val) syntactically identical — both parse to a bare field_expression/Term.Select with no argument list. Resolving those by simple name would turn a same-named field read into a spurious CALLS edge, so neither cgr's call extractor nor the oracle names a bare select; both grade Term.Apply/ApplyInfix (and cgr's call_expression/infix_expression) sites, which carry an explicit application. Measured on the two corpora, resolving bare selects added zero edges anyway (the enclosing applications already cover the real calls), so this costs no recall. The residual is the diffuse receiver-type-inference tail every language eval carries (implicit conversions such as "" ~>, deeply generic receivers), not a systematic gap.

The same harness applied to C#: for each first-party C# symbol, which files call it. cgr's C# CALLS and INSTANTIATES edges, reduced to (caller_file, callee_simple_name), are graded against invocation and object-creation sites extracted by Roslyn (the same oracle as the C# L1 structure eval), over the same first-party name universe. INSTANTIATES counts because new T() on a type with no explicit constructor has no ctor node to CALL — the oracle records the creation site by type name either way, and class names join the declared universe so a creation of a fully ctorless type is graded too. Roslyn's parser is independent of cgr's tree-sitter C# frontend.

uv run python -m evals.csharp_retrieval --target <csharp-sources>
# opt-in Roslyn semantic frontend (issue #738):
CSHARP_FRONTEND=hybrid uv run python -m evals.csharp_retrieval --target <csharp-sources>

Requires the dotnet toolchain on PATH; the eval exits cleanly if it is missing. Pinned by codebase_rag/tests/test_csharp_retrieval_eval.py, where cgr's C# call graph matches the Roslyn oracle on the fixture.

This is the eval that measures the #738 hybrid frontend's win. On Polly (~600 files), precision is 1.0 in both modes and the hybrid frontend lifts recall 0.774 → 0.895 (F1 0.873 → 0.945): the location-keyed Roslyn call facts resolve the overloads, extension methods, and cross-project dispatch the tree-sitter heuristics cannot. The residual misses are dominated by BCL name collisions the simple-name universe cannot distinguish (list.Add graded against a first-party Add), plus call sites in projects outside the target's solution file (samples/, bench/), which the frontend deliberately degrades to heuristics for.

cgr's semantic search embeds each function's source and retrieves by cosine similarity to a query embedding. This grades that relevance directly: for controlled fixtures whose natural-language query maps unambiguously to one function, does cgr's embedder rank that function in the top k?

It uses cgr's own embedder over function source extracted from the captured graph (the same text cgr embeds), computes the ranking, and scores recall@k against curated query -> function cases (e.g. "read and parse a json file" should retrieve load_json_file, not send_email or compute_sales_tax). This tests the embedding-and-ranking pipeline that decides relevance; the Qdrant ANN layer only approximates the same cosine ranking, so it is out of the loop here.

Requires the semantic extra (embedding model); the eval is skipped when it is absent. Pinned by codebase_rag/tests/test_semantic_search_eval.py, where cgr reaches recall@3 = 1.0 on the fixture. The relevance set is curated and deliberately clear-cut: this is a regression guard that the pipeline retrieves obviously-relevant code, not a broad relevance benchmark (which would need a large human-judged dataset).

The Python L1 above grades cgr against a Python ast oracle. To grade other languages with independent ground truth, each language is checked against its own standard-library parser rather than against cgr's own tree-sitter output. The first such oracle is Go.

uv run python -m evals.go_l1 --target /path/to/go/repo --project-name myrepo

How it works:

• Oracle (evals/oracles/go_ast.go): a small Go program that walks the target with the standard library's go/parser + go/ast and emits one JSON record per declaration (function-local type declarations included, via ast.Inspect, since cgr captures those too). The kind field already uses cgr's NodeLabel vocabulary (Function, Method, Class, Interface, Type), so records join cgr's nodes directly on (kind, file, start_line). Mapping: func → Function, func with a receiver → Method, type … struct → Class, type … interface → Interface, any other type … (defined types and aliases) → Type. Requires the go toolchain on PATH; evals.go_l1 exits cleanly if it is missing.
• cgr side (cgr_graph.extract_cgr_go_nodes): builds cgr's graph over the target and keeps the Go (.go) definition nodes.
• Fair file set: run_go_oracle drops oracle records under any directory in cgr's IGNORE_PATTERNS (e.g. bin, vendor, build), so the oracle grades exactly the files cgr indexes — single source of truth, no drift.
• Score: per-kind precision/recall/F1 via score.score_node_kinds, written to evals/results/go_scores.csv and evals/results/go_diff.json.

Validated on apache/thrift (1604 cgr Go nodes vs 1604 oracle nodes — exact):

Both gaps the oracle originally exposed are fixed: Go type declarations (struct/interface/defined-type) are captured (see codebase_rag/tests/test_go_type_declarations.py), and Go receiver methods are now Method nodes qualified by their receiver type with a DEFINES_METHOD edge from it (see codebase_rag/tests/test_go_receiver_methods.py), rather than being mislabelled Function.

The second native oracle is Rust, checked against syn (the de-facto standard Rust parser).

uv run python -m evals.rust_l1 --target /path/to/rust/repo --project-name myrepo

• Oracle (evals/oracles/rs_oracle/): a small Rust program that parses every .rs file with syn and emits one JSON record per declaration, in cgr's NodeLabel vocabulary. A syn::visit::Visit walk recurses into function bodies (function-local defs), impl/trait associated types, and closures (which cgr models as anonymous Function nodes), so the comparison is apples-to-apples. Mapping: struct → Class, enum → Enum, union → Union, trait → Interface (+ its methods → Method), type (incl. associated types) → Type, fn/closure → Function, impl method → Method. Requires the cargo toolchain (proc-macro2's span-locations feature gives real line numbers); evals.rust_l1 exits cleanly if it is missing.
• cgr side (cgr_graph.extract_cgr_rust_nodes), score (score.score_node_kinds), output to rs_scores.csv / rs_diff.json.

Validated on apache/thrift's lib/rs (758 cgr Rust nodes vs 758 oracle nodes — exact, all kinds 1.0). The oracle surfaced one cgr gap, now fixed: methods in an impl Trait for <primitive> block (e.g. impl From<Foo> for u8) were dropped because the primitive_type impl target was unhandled (see codebase_rag/tests/test_rust_impl_primitive_target.py).

The third native oracle is TypeScript, checked against the TypeScript compiler API.

uv run python -m evals.ts_l1 --target /path/to/ts/repo --project-name myrepo

• Oracle (evals/oracles/ts_oracle/): a Node script that parses every .ts/.tsx file (.d.ts excluded) with the TypeScript compiler API and emits one JSON record per declaration, in cgr's NodeLabel vocabulary. Mapping, matching how cgr models TypeScript: class → Class, interface → Interface, enum → Enum, type → Type, namespace/module → Class (a class-like container), function → Function (or Method inside a namespace/class), arrow functions and function expressions → Function (cgr captures every one, like a Rust closure), method/constructor → Method. Requires node/npm (the typescript dependency is installed on first run; package-lock.json is committed and node_modules/ is gitignored). evals.ts_l1 exits cleanly if node is missing.
• cgr side (cgr_graph.extract_cgr_ts_nodes), score (score.score_node_kinds), output to ts_scores.csv / ts_diff.json.

Validated on apache/thrift's TypeScript (lib/nodets, lib/ts): 136 cgr nodes vs 136 oracle nodes — exact, all kinds 1.0. No cgr gap found.

The same compiler-API oracle parses JavaScript too (the TypeScript compiler accepts JS), so JavaScript reuses evals/oracles/ts_oracle/ over .js/.jsx.

uv run python -m evals.js_l1 --target /path/to/js/repo --project-name myrepo

Same mapping as TypeScript, with two JS-specific points matching cgr: object-literal shorthand methods are modelled as standalone Functions (not Methods), and every arrow function / function expression is a Function. Output to js_scores.csv / js_diff.json.

Validated on apache/thrift's JavaScript (lib/js, lib/nodejs): 1087 cgr nodes vs 1087 oracle nodes — exact, all kinds 1.0. No cgr gap found.

The sixth native oracle is Java, checked against the JDK's own parser (com.sun.source / javax.tools).

uv run python -m evals.java_l1 --target /path/to/java/repo --project-name myrepo

• Oracle (evals/oracles/java_oracle/Oracle.java): parses every .java file with the JDK Compiler Tree API (task.parse() only parses, so missing dependencies are fine) and emits one JSON record per declaration. Mapping, matching how cgr models Java: class → Class, interface → Interface (+ its method signatures → Method), annotation type (@interface) → Class, enum → Enum, method/constructor → Method. A method declared inside an anonymous class (e.g. new Runnable() { public void run() {...} }) is modelled as a standalone Function — the same way cgr treats it (and JS object-literal methods); the oracle replicates cgr's rule (a member is a Method only when its nearest enclosing named class precedes any enclosing method/lambda body). Requires javac/java; the oracle is compiled on first run (the .class is gitignored, the source committed). evals.java_l1 exits cleanly if the JDK is missing.
• cgr side (cgr_graph.extract_cgr_java_nodes), score (score.score_node_kinds), output to java_scores.csv / java_diff.json.

Validated on apache/thrift's lib/java: 2861 cgr nodes vs 2861 oracle nodes — exact, all kinds 1.0 (including the 103 anonymous-class methods graded as Function). No cgr gap found.

The seventh native oracle is Lua, checked against luaparse.

uv run python -m evals.lua_l1 --target /path/to/lua/repo --project-name myrepo

• Oracle (evals/oracles/lua_oracle/): a Node script that parses every .lua file with luaparse (luaVersion: "5.3", so bitwise operators / integer division parse) and emits a Function record per function declaration/expression. Lua has no classes, so cgr models every function — global, local, table (t.f), method (t:m), and anonymous function expressions — as a Function. Requires node/npm (the luaparse dependency installs on first run; package-lock.json committed, node_modules/ gitignored).
• cgr side (cgr_graph.extract_cgr_lua_nodes), score (score.score_node_kinds), output to lua_scores.csv / lua_diff.json.

Validated on apache/thrift's Lua (lib/lua, test/lua): 376 cgr nodes vs 376 oracle nodes — exact, 1.0. No cgr gap found.

The eighth native oracle is PHP, checked against php-parser (a pure-JS PHP parser, so no php binary is needed).

uv run python -m evals.php_l1 --target /path/to/php/repo --project-name myrepo

• Oracle (evals/oracles/php_oracle/): a Node script that parses every .php file with php-parser and emits one record per declaration. Mapping, matching cgr: class → Class, interface → Interface (+ methods → Method), trait → Class (+ methods → Method), enum → Enum, function → Function, closure / arrow fn → Function. Methods of an anonymous class (new class {...}) are Functions (like Java/JS object-literal members), and a declaration's line is its first attribute (#[Attr]) line when present — both matching cgr's node span. Requires node/npm (the php-parser dependency installs on first run; package-lock.json committed, node_modules/ gitignored).
• cgr side (cgr_graph.extract_cgr_php_nodes), score (score.score_node_kinds), output to php_scores.csv / php_diff.json.

Validated on apache/thrift's PHP (lib/php): 1295 cgr nodes vs 1295 oracle nodes — exact, all kinds 1.0. No cgr gap found.

The ninth native oracle is C#, checked against Roslyn's own syntax parser (independent of cgr's tree-sitter frontend and of the opt-in Roslyn semantic frontend).

uv run python -m evals.csharp_l1 --target /path/to/csharp/repo --project-name myrepo

• Oracle (evals/oracles/csharp_oracle/): a .NET program parsing every .cs file with CSharpSyntaxTree and emitting nodes (Class/Interface/Enum/Method/Function), containment (DEFINES, DEFINES_METHOD), and inheritance name-edges. Conventions match cgr's model: interface bases are INHERITS, class-declared interface bases are IMPLEMENTS, top-level script functions anchor to the file Module, and #if/#elif/#else directives are neutralized so every branch parses (cgr has no preprocessor and indexes all branches).
• Phantom-member suppression (issue #768): an expression-bodied member whose body is split across #if/#else has two bodies once the directives are neutralized, which is ill-formed C#; Roslyn ends the member at the first branch's ; and error-recovers the second branch's expression as a phantom member declaration. Members carrying parse diagnostics are dropped — real repos compile, so genuine members are diagnostic-free. Calls inside every branch still count (the walk visits the recovered expression's descendants).
• Known residual: on such split members the two parsers' recoveries legitimately disagree — tree-sitter stretches the member's span across both branches and can swallow the next member's attribute into the garbled region. On Polly this leaves 1 member start-line disagreement and 2 span mismatches out of 6,551 nodes; neither view of the ill-formed neutralized source is "correct", so this is documented rather than bent to either parser.
• cgr side (cgr_graph.extract_cgr_csharp_graph), output to csharp_scores.csv / csharp_diff.json.

Validated on App-vNext/Polly (~6,550 nodes): nodes/DEFINES/DEFINES_METHOD 0.9998, INHERITS and IMPLEMENTS exact 1.0 (224 and 147 edges, same-scope arity pairs included per #764), spans 0.9997. The residual is exactly the documented preprocessor-split artifact.

Committed snapshots live in evals/results/ — scores.csv (L1), diff.json (L1 per-label missing/extra), calls_diff.json (L3 missed edges). Regenerate with the commands above.

Span (end_line) accuracy on matched defs: 6800/6800 exact.

The L3 fixture exercises rich Python plus all 11 supported languages; recall is a sound lower bound over the cgr code paths that fixture drives. These numbers are for the Python codebase_rag target — graded multi-language recall (JS/Rust/Go/Java/C/C++/Lua/PHP/Scala) is future work pending a SCIP-based oracle.

The resolved graph more than doubles the precision of even the call-tuned grep (0.85 versus 0.37) at near-perfect recall, for an F1 of 0.91 versus 0.54: a gain of roughly 0.38 absolute (about 70% relative) over the strongest grep baseline. Bare-name grep, the common first attempt, scores far worse (F1 0.38). This is the decoupled retrieval result behind the intuition that a structural graph beats text search for code navigation.

Over a 25-file neutral-edit sample on codebase_rag, after the #532 fix (micro-averaged across probes; clean re-index is the oracle, so fn is edges the incremental graph dropped and fp is stale edges it kept):

10 of 25 edits now reproduce a clean re-index exactly (up from 3 before the fix), INSTANTIATES is perfect, and IMPORTS is all but perfect. For reference, before the fix the same sample showed CALLS fp/fn of 4/3318, IMPORTS 7/599, INSTANTIATES 0/414, and only 3/25 clean-equivalent: the fix cut CALLS divergence by roughly 75% and IMPORTS by 98%. The residual is the changed file's own calls resolved through type inference / protocol dispatch (the fp/fn are mostly the same call resolved to the protocol method incrementally versus the concrete implementation in a clean pass), which needs full cross-file type context to close (see the methodology note above). Tracked under issue #532.

A clean negative result: on codebase_rag, cgr classifies every import correctly, internal and external alike. This rules out #498-style misclassification on this corpus and stands as a regression guard. (The first run reported 247 missing externals; investigation showed they were all from __future__ import ..., an oracle over-count now corrected rather than a cgr bug.)

Another clean negative result within the graded scope (top-level classes; single-inheritance for overrides): cgr resolves every base to the correct first-party class and attributes every single-inheritance override to the right base. The first run showed minor fp/fn; investigation traced all of them to oracle scope (a class nested in a test method, and multi-base mixin classes whose override attribution is MRO-decided), not cgr defects, so the scope was tightened rather than the discrepancies reported.

cgr localizes every constructor call exactly on codebase_rag: the INSTANTIATES set and the ast oracle's constructor-call set are identical.

The above isolates retrieval. The complementary end-to-end measurement is GKG's own design: hold one agent and model fixed and vary only the tools (graph tools versus grep), then report SWE-bench-style resolved rate over real issues. That needs an LLM, a container harness, and many runs, so it is tracked separately rather than run inside this deterministic harness.