RingFacts · 9 August 2026

What the tests cover, and what they deliberately don’t

Three tiers, split by what each one needs rather than by what it is called. That single distinction is the whole design: this repository commits to itself, unattended, every six hours, so the tests that gate a commit have to run with no network, no API keys, and no database.

95checks on every commit
0.43sto run them
19more against real SQL
0credentials required

The one idea

Everything else follows from where the gate falls.

Before this, verification was four scripts at the repository root, each written the day something broke. None ran together, most needed live credentials, and nothing failed when they did. That was survivable while every commit was typed by hand.

It stopped being survivable once the check-in runs began committing code at 4am. self-improvement.md §4 tells those runs to verify before claiming — but a principle nothing enforces is a principle that erodes. So the tests that gate a commit must be runnable by an agent with nothing mounted.

pre-commit gate — blocks the commit if anything below fails
Tier 1 Unit & fixtureDo the pure functions compute the right answer? nothing
Tier 2 PipelineAre those functions plugged together correctly? nothing
Tier 3 Real SQLDoes Postgres agree with what the code believes? a Neon branch
— Matcher evalHow often does the LLM get it right? (not built yet) not a test

The gate covers the two tiers that need nothing. Tier 3 skips itself when TEST_DATABASE_URL is unset, so it never blocks a session that has no secrets. The fourth row is not a test at all — see below.

The tiers

What each one is responsible for, and how it avoids the network.

Tier 1 — Unit & fixture

64 checks · lib/*.test.js
Question
Given this input, does this one function return the right thing?
Covers
  • The outlet name filter — the highest-stakes function in the repo. Feeds are filtered before storage, so a dead surname stem drops real coverage leaving no row behind. No query can tell that apart from quiet news.
  • Verdict validation — every way a model answer can be malformed, and the downgrade each one earns.
  • The five-rung extraction ladder — which rung fires, and the paragraph-doubling pathology that once inflated a mention count.
  • The digest tier rule — its boundary, plus the three “we cannot tell” escapes that must never demote.
Fixtures
Reduced structural skeletons, not saved article pages — small enough to read, and no republishing of other people’s articles into a public repo. Each names the pathology it encodes.
✓ a Cyrillic stem matches declined forms
✓ a name in an image alt attribute does not match
✓ a numeric id from the model matches a string id from Postgres
✓ a body too short to judge is never demoted

Tier 2 — Pipeline

30 checks · test/pipeline.test.js
Question
Feed real articles through the real hunt — what posted, what was held, and why?
How
The hunt function gained one optional deps parameter whose every default is the real implementation, so production behaviour is unchanged. Tests swap in a scripted matcher, a scripted embedder, and an in-memory store. Articles are handed in directly, so no feed is ever fetched.
Covers
  • Both duplicate gates — the URL gate under both identities, and the 0.80 semantic threshold with its official-source exemption and the re-application that stops a matcher outage turning every official echo into a repeat post.
  • The digest tier split, and the suppression branch that corrects rows back to unposted so later threshold measurements stay honest.
  • Claim lifecycle: rumour born, official confirmation, echo held as evidence.
  • Every fail-open path — asserted in the docs for months, verified by nothing until now.
Fidelity
The fake store returns ids as strings, because every id column is bigint and Postgres does. A fake that returned numbers would hide the exact bug that nearly shipped last week.
✓ an embedding outage degrades to URL-only dedup and still posts
✓ a matcher outage becomes UNSURE and the article posts as it always did
✓ when everything was tangential, nothing is broadcast at all
✓ suppressed rows are corrected to posted=false so the audit stays honest

Tier 3 — Real SQL

19 checks · test/sql.test.js · opt-in
Question
Does the actual database behave the way the code assumes?
Why separate
A fake can never check pgvector’s cosine arithmetic, whether ON CONFLICT returns null instead of throwing, or whether the schema still has the columns the code writes. Those need real Postgres.
Safety
Runs against a Neon branch — copy-on-write off main, so it carries the real schema and real data shape without touching the evidence record. It refuses to run against main, tags every row it writes with a per-run key, and removes them afterwards.
✓ knownUrls matches on url and on resolved_url alike
✓ similarity is 1 − cosine distance, and the nearest row wins
✓ confirmClaim flips a rumor exactly once
✓ the embedding column’s width matches what the embedder requests

The matcher — deliberately not tested

designed, not built
Why
Four runs of byte-identical code on the same article returned NO_CLAIM three times and NEW once. Any assertion built on that flakes, and a suite you learn to ignore is worse than no suite at all. So the matcher is stubbed in every tier above.
Instead
A separate eval: roughly thirty labelled articles from the archive, each run several times, scored as a pass rate with spread rather than pass or fail. Run by hand when the prompt changes; never on commit.
First job
Settle a question already sitting open in the backlog — run the corpus at the default sampling temperature and at zero, compare the rates, and the decision stops being a guess.

What it caught on the way in

Four real defects, none of which were found by reasoning about the code.

found by — a dry run against live feeds

Every non-English headline was posting untranslated

The new dependency object was written with translateToEnglish as JavaScript shorthand, so its key was translateToEnglish while the call site read deps.translate. Translation failed on every foreign-language article — and because that path fails open, the original headline posted and nothing looked wrong.

All thirty pipeline tests passed, because each one supplies its own dependencies and none exercised the defaults. The suite now checks its own wiring at the source level, and that check has been verified to fail when the bug is put back.

found by — mutating the thresholds on purpose

The freshness window had no coverage at all

Changing the duplicate threshold broke three tests; changing the tier rule broke three more. Changing how far back the hunt looks broke nothing — every test pinned that value explicitly, so the filter itself was never exercised. Two tests added.

found by — tier 3, on its first run

One number lived in two places with nothing connecting them

The schema declares a 768-wide vector column; the embedder asked the API for 768 as a private constant. Changing either alone would have failed every insert at runtime in the cloud, with no local signal. The constant is now shared, and the test asserts the columns match it.

found by — reading the files being committed

A real watchlist name was about to re-enter the public repo

Two tracked files still named an actual tracked subject — the thing a commit last week set out to remove. Both now use a synthetic name, as the fixtures do.

Running them

The first command is the one that matters.

# every commit runs this automatically; 0.43s, offline
npm test

# once per clone, to enable the gate
git config core.hooksPath .githooks

# the SQL tier, against a branch — never main
TEST_DATABASE_URL=$(neonctl connection-string test \
  --project-id calm-mouse-60802247) npm run test:sql

# still required before deploying — stubs cannot see real wiring
DRY_RUN=1 node hunter.js
The tests do not replace the dry run. They stub the network on purpose, which means they cannot see whether the real parts are plugged into each other. Stubs verify logic; only a real run verifies wiring — which is exactly how the translation bug was caught after the whole suite went green.

What this deliberately leaves out

Each omission is a decision, not an oversight.

No CI. The commit hook is the gate. Deployment runs from setup.sh, not a pipeline, so a hosted workflow would test code that is already committed and add a second place for the rules to live.

No coverage threshold. A percentage target rewards testing whatever is cheapest to test — here that would be the message formatters, not the name filter that can silently lose an entire outlet.

No mocking library. The seams are ordinary function parameters with real defaults. Nothing is patched at runtime.

The audit scripts stay as they are. They measure production data to tune thresholds. That is instrumentation, not testing, and merging the two would break the measurement discipline the thresholds depend on.