Every benchmark/diagnostic entry point in the repo. Benchmarks mostly measure the properties the architecture is built for: honest abstention, held-out generali

Benchmarks & Diagnostics

Every benchmark/diagnostic entry point in the repo. Benchmarks mostly measure the properties the architecture is built for: honest abstention, held-out generalization, cross-domain transfer, catastrophic-forgetting resistance, and graph latency.

All commands are run from the repo root. scripts/ is on the path for scripts.X imports; the src dirs are auto-prepended.

Comprehensive evaluation harness (vs nanoGPT)

scripts/evaluate_ravana.py is the primary end-to-end harness. It trains RAVANA on Tiny Shakespeare once, snapshots the weights, then runs every benchmark in isolation (fresh engine per category, so benchmarks cannot leak state into each other) and prints a RAVANA-vs-nanoGPT comparison.

# Full run (all benchmarks, fresh 25-pass Shakespeare training)
python scripts/evaluate_ravana.py

# Skip retraining if a snapshot already exists
python scripts/evaluate_ravana.py --skip-train

# Disable live-web curiosity loops (faster, deterministic; recommended for CI)
python scripts/evaluate_ravana.py --no-curiosity

# Enable the learned subsystems that replace the default hardcoded backstops
python scripts/evaluate_ravana.py --source-trust --learned-pos --intent-router

# Opt-in semantic grading for LoCoMo/LongMemEval/TimeDial/MemFail
# (reuses RAVANA's own GloVe vectors; fail-open — exact-substring otherwise)
python scripts/evaluate_ravana.py --semantic-grade

# Limit cases per benchmark / choose a subset
python scripts/evaluate_ravana.py --max-cases 20 --benchmarks lamp_test,reasoning

Benchmarks run by default (key → dataset):

Key	Dataset	What it measures
`lamp_test`	synthetic 3-premise causal chain	causal reasoning (no explosion)
`reasoning`	LogiQA (8,678 QA)	logical reading comprehension
`temporal`	TimeDial (1,446 cloze)	temporal commonsense in dialog
`locomo`	LoCoMo (10 convs, 1,986 QA)	long-term conversational memory
`long_mem_eval`	LongMemEval (500 oracle QA)	multi-session extraction, temporal, KU, abstention
`adversarial`	AdvBench (520 items)	harmful-behavior refusal
`memory_consistency`	MemFail	coexisting/conditional facts, long-hop, persona
`self_evaluation`	synthetic	meta-cognition / honest self-knowledge
`consult`	synthetic	advice/wellbeing support

Results (params, per-benchmark scores, overall average) are written to data/eval_results.json. The current live run's numbers are summarized in README.md → Benchmark results. completes; the harness prints the full RAVANA-vs-nanoGPT table at the end of every run.

Chat / end-to-end

Script	Measures
`scripts/ravana_chat.py`	The live chatbot. Not a benchmark, but the reference runtime.
`scripts/benchmark_queries.py`	6-class query taxonomy (chitchat, factual, hypothetical, conditional, identity, OOD). `OOD` items are abstention probes — a confident answer is a confabulation. Run `python -m experiments.benchmark_arc` to drive it.

Decoder / relation learner (RLMv2)

Script	Measures
`scripts/benchmark_vs_transformers.py`	P3 harness: verb-offset held-out generalization, cross-domain transfer, ontology benefit, catastrophic forgetting, conversation quality, parameter efficiency. `-p scripts/benchmark_vs_transformers.py --quick`.
`scripts/external_benchmark.py`	PCX/NeuroBench-style text tasks, lifelong retention under task-switching, large-graph (~100–200K node) latency/memory profiling. `--quick` for a fast run.
`scripts/diagnose_transfer.py`	Cross-domain semantic transfer & held-out diagnostic (Science→Social) with verb-offset + test-time adapter adaptation.
`scripts/validate_held_out_generalization.py`	Phase-1 P0 checks: all-verb offset support, confidence-weighted blending, prototype inheritance, cross-domain transfer.
`scripts/triple_eval.py`	Per-triple diagnostics (relation type, PE, confidence, source, edge attributes). `--triples-file` or interactive.
`scripts/run_ablation.py`	Off-topic-snippet rate for paradox grounding, before vs after the gated retrieval fix.
`scripts/scaling_benchmark.py`	`ConceptGraph` latency at 1K→10K nodes.

Analysis / profiling helpers

Script	Purpose
`scripts/profile_phase3.py`	`cProfile` of `RLM.learn()` + individual graph-op timings.
`scripts/plot_analysis.py`	Post-hoc plots (backward transfer, concept drift, cross-domain summary) from `experiment_results/`.
`scripts/analyze_longitudinal.py`	Reads `checkpoints/longitudinal/metrics.jsonl`; phase-transition / forgetting analysis.
`scripts/audit_coverage.py`	Compares source modules against test imports (coverage audit).

Corpus / artifact setup (one-shot)

Script	Purpose
`scripts/gather_teen_seeds.py`	Fetch teen-level conversational English → `data/corpora/teen_seeds.txt`.
`scripts/expand_corpus.py`	Expand the corpus via the IntentForge gateway.
`scripts/train_lancaster_probe.py`	Train the wide-coverage Lancaster sensorimotor probe → `data/lancaster_encoder.npz`.

Experiments package (`experiments/`)

Research harnesses behind the numbers above: experiment_cross_domain.py (build_domain_a_science / build_domain_b_social), benchmark_arc.py (ALL_QUERIES), experiment_ablation.py, experiment_sleep_memory.py, experiment_chat_quality.py, etc. Imported by the benchmark scripts; not run directly in normal use.

Interpreting results

Abstention probes (OOD class) must score below honest "I don't know". Confident-wrong is treated as high free-energy.
Held-out / cross-domain top-1 is expected to be modest; top-10 and verb-offset behavior are the meaningful signals.
Forgetting: with sleep consolidation, Domain-A retention after Domain-B training should stay well above the ~14% random baseline.

Benchmarks & Diagnostics

On this page