NetClaw has 227 skills now. Each one is a Markdown file telling an agent when to reach for it, and at 227 of them I couldn’t keep dodging the question: does anyone actually know if these are still good? Reading 227 files by hand is tedious but doable. Reading all 25,651 possible pairs of them for overlap is not. So I handed the job to a model I’d never used before — Jev, from a company called TypeSafe — and spent 11,892 of its answers finding out.
What Jev actually is
If you’ve spent any time with an LLM, you know the pattern: ask a question, get back a paragraph, then write code to parse that paragraph back into the yes/no or the category your program actually needed. TypeSafe built something for the step that comes right after the parsing — a model that skips the prose entirely.
Jev is TypeSafe’s flagship “System One” model, named after the fast, intuitive “System 1” thinking Daniel Kahneman described in Thinking, Fast and Slow. You send it a state — any text or JSON — plus a set of typed questions, and it evaluates all of them in parallel against that same state, returning calibrated probabilities rather than generated text, in roughly 100–500ms per call. There are three question types:
- Noul — a yes/no question. Returns the probability of “yes.”
- Choice — pick one of N labeled options. Returns the full probability distribution.
- Score — a graded rating along a rubric you define.
That’s the whole interface. Ask a batch of narrow, independent questions over the same context, get back numbers, and let your own code decide what to do with them. At the pricing in effect for this project it’s $0.042 per million input tokens — and output is free outright, not just cheap.
That’s not a promotional discount. It falls out of how the model is trained. TypeSafe calls its approach RLCD — Reinforcement Learning for Calibrated Decisions — a third path they draw alongside RLHF (the technique behind ChatGPT) and RLVR (behind today’s reasoning models). The pitch is specific: RLHF optimizes a model to say things people prefer, which is also what makes chatbots sycophantic and confidently wrong. RLCD optimizes for something else entirely — that a probability of 0.8 should be right about 80% of the time, across many predictions. Jev never generates text to get there. There’s no token-by-token sampling of a response to meter; an answer is a probability, a chosen label, or a graded score, computed in one pass over the input. The bill is for reading, not writing — you get a probability back, not a paragraph.
Three sweeps, cheapest first
All the harness code lives in NetClaw’s repo under scripts/jev-audit/ — throwaway tooling, not a permanent part of the product. Three sweeps:
| Sweep | What it asked | Calls |
|---|---|---|
| 1. Skill health | One call per skill, full SKILL.md as context, four Noul questions together: ambiguous trigger, no failure behavior specified, stale machine/path assumptions, prompt-injection smell |
227 |
| 2. Overlap matrix | Every one of the 25,651 possible skill pairs, checked for whether they claim the same trigger territory | 630 |
| 3. Coverage gaps | 40 candidate skills + 20 candidate MCP servers I generated, each checked against all 227 existing skills to confirm it was actually missing | 300 |
Sweep 2 is the one worth explaining, because the naive version of it doesn’t work. A pairwise check across 227 skills is C(227,2) = 25,651 pairs. One call per pair is 25,651 calls — which ignores TypeSafe’s own guidance that adding a question to a call is nearly free but adding a call is not. So instead, each skill’s description became one call’s context, with one short question about every skill after it in a sorted list. Same 25,651 pairs covered, 630 calls.
I hit two real bugs building that batching, and both are worth keeping. The first version repeated a long boilerplate paragraph inside every single question — the largest anchor’s request came in around 58,000 tokens, right at Jev’s 64k context ceiling, and the request stalled outbound with no exception and no timeout ever firing. Fixed by stating the rubric once in state instead of once per question, and hard-capping every call at 50 comparisons. The second bug was sneakier: self-consistency re-checks (more on those below) were running sequentially inside each batched call’s own thread, so a chunk with a lot of borderline answers could serialize up to 250 sequential calls before that one job ever reported done — which looks exactly like a stall if you’re only watching a progress counter. Splitting the primary pass and the re-check pass into two independently-parallelized phases fixed it.
The bill, polled straight from the API
Every number below is summed from the usage field TypeSafe actually returned on each of the 11,892 calls, not an estimate. I set a stop threshold of $1.50 per sweep against a $5 starter budget before running anything.
| Sweep | Calls | Input tokens | Cost |
|---|---|---|---|
| 1: skill health | 462 | 1,287,934 | $0.0541 |
| 2: overlap matrix | 10,900 | 8,497,040 | $0.3569 |
| 3: coverage gaps | 530 | 1,671,145 | $0.0702 |
| Total | 11,892 | 11,456,119 | $0.4812 |
Most of that spend went to self-consistency checks. When a probability landed in the uncertain 0.30–0.70 band, the harness re-ran that exact question five more times with a fresh throwaway ID and only trusted the result if the five answers agreed. Of sweep 2’s 10,900 calls, 10,270 were re-checks. That’s the most important design decision in the whole project: a probability is not the same thing as ground truth, and treating a wobbly answer as a lower-confidence maybe rather than a fact is the difference between an audit and a slot machine.
What it actually found
- 123 of 227 skills document no failure behavior — no guidance for what the agent should do if a tool call errors or a credential is missing.
- 336 of 25,651 pairs share real trigger territory at noul ≥ 0.6, not just surface keyword overlap.
- 8 of 227 skills were flagged for stale, machine-specific assumptions. 3 turned out to be real; 5 were false positives on closer read.
- 47 of 60 candidate skills/MCPs I proposed were confirmed by Jev as genuinely uncovered elsewhere in NetClaw.
The single highest-confidence overlap in the whole matrix, at 0.97, was between document-generation and network-report-documents — a generic document-building engine and four canned reports built on top of it, described almost identically. Right behind it at 0.93: two entirely separate, independently-built MCP server implementations for talking to Cisco Catalyst Center, wired up to four different skills between them, that nobody had noticed were solving the same problem twice.
The bug a probability score couldn’t explain
Jev flagged memory and mempalace — NetClaw’s own memory system and an alternate community tool — as overlapping at 0.85. Going in to write a disambiguating note for both descriptions, the actual cause turned up: memory/SKILL.md had no YAML frontmatter at all.
- # Skill: Persistent Memory
+ ---
+ name: memory
+ description: "NetClaw's native persistent memory (spec 033) —…"
+ ---
+
+ # Skill: Persistent MemoryJev’s description-extractor had nothing to read for that skill and effectively saw an empty entry — a completely different failure mode than “these two skills are actually redundant.” A noul score has no way to distinguish those two explanations from each other; it can only tell you the two skills read as similar. Reading the actual files is what turns “these look alike” into “here’s why, and here’s the fix.”
That’s the shape of the whole project, really. Jev is what makes it economically sane to ask the same narrow question 11,892 times — it graded 227 skills against a fixed rubric in under a minute for about five cents, and answered every single one of 25,651 pairs rather than a sample of them, for thirty-six cents. What it can’t do is tell a real bug from a false positive, or decide what to actually change. Of the 8 stale-assumption flags, 5 were correctly left alone once I actually read the files; of the top 26 overlapping pairs, 5 were legitimate design (a genuine discover→design→validate pipeline, not redundancy) and 21 got real fixes. TypeSafe’s own docs put it plainly: typed output guarantees the interface, not truth. Jev is the triage. Reading the flagged file is still the job.
What I didn’t fix
123 skills got a tailored failure-behavior section — generated per skill from its own referenced environment variables and whether it has write-capable tools, not one paragraph pasted 123 times. 3 real stale-machine bugs got fixed (a Cisco DevNet sandbox hostname baked into a skill, my own name baked into a federation-member path, a hardcoded Twitter handle). 21 of the 26 highest-confidence overlapping pairs got disambiguated.
Two things are still sitting there on purpose. The 310 lower-confidence overlap pairs, between 0.60 and 0.85, haven’t been reviewed — only the top 26 were. And the 47 confirmed coverage gaps — things like Cisco SD-WAN vManage write operations, Aruba ClearPass NAC, Microsoft Sentinel, HashiCorp Consul — are not new skills yet. Jev confirming a gap is real is not the same thing as having credentials to test against a real Sentinel tenant or a real ClearPass appliance. Fabricating 47 integrations against real vendor APIs with no testing behind them would just be untested code pretending to work, and NetClaw has enough of those phantom-tool bugs in its history already without me adding new ones on purpose. Each of those 47 becomes its own future spec, the same way everything else in this repo does.
Where this goes next
NetClaw’s skill count only goes up. The obvious next move isn’t a one-off audit — it’s re-running sweep 1 and 2 on a schedule, or whenever a new skill lands, so a colliding trigger gets caught before it ships instead of found later in a pairwise sweep of everything at once. And the 310 unreviewed pairs and 47 confirmed gaps are sitting right there as a backlog, already ranked by confidence, waiting to become the next set of specs.
The wider idea is the one TypeSafe’s own site keeps coming back to: anywhere an agent currently makes a judgment call without a calibrated probability behind it — ticket routing, alert triage, picking which of several similar tools to call — is a candidate for the same treatment. Forty-eight cents is a genuinely strange price to have paid to find out this much about my own project.
Sources: typesafe.ai, docs.typesafe.ai/concepts/system-one, docs.typesafe.ai/primitives, docs.typesafe.ai/api, docs.typesafe.ai/models, docs.typesafe.ai/introduction/machine-learning-primer — the RLCD explanation, typesafe.ai/manifesto, and NetClaw PR #264, where every fix and every diff from this project is public.