Merge pull request 'Jev research: overview, pi-harness use cases, dreaming method' (#1) from feat/jev-research into main
Reviewed-on: #1
This commit is contained in:
commit
7c749a6eee
211
docs/01-what-is-jev.md
Normal file
211
docs/01-what-is-jev.md
Normal file
@ -0,0 +1,211 @@
|
|||||||
|
# Jev: what it does and how it works
|
||||||
|
|
||||||
|
Research summary of TypeSafe AI's Jev, launched 15 September 2026. Written so an
|
||||||
|
undergraduate with one AI course behind them can follow it. Sources at the
|
||||||
|
bottom; claims are tagged confirmed, self-reported, or speculative.
|
||||||
|
|
||||||
|
## TL;DR
|
||||||
|
|
||||||
|
Jev is not a chatbot and not a smaller LLM. It is a **decision model**: you send
|
||||||
|
it a piece of state (text or JSON) plus a set of pre-defined questions, and it
|
||||||
|
returns typed answers — a category, a number on a scale, or a probability — each
|
||||||
|
with calibrated probabilities. It never generates text. That single design
|
||||||
|
choice makes it ~100x faster and ~100-400x cheaper than a frontier LLM on these
|
||||||
|
narrow questions, and makes malformed answers structurally impossible.
|
||||||
|
|
||||||
|
## The problem it solves
|
||||||
|
|
||||||
|
A normal LLM returns **strings**. To use a string inside software you must
|
||||||
|
parse it, validate it, and hope the model did not go off the rails. Three costs
|
||||||
|
follow:
|
||||||
|
|
||||||
|
1. **Latency.** LLMs generate one token at a time, each conditioned on the
|
||||||
|
last. A 500-token answer is ~500 sequential forward passes. Frontier
|
||||||
|
responses take 3s to 329s end to end.
|
||||||
|
2. **Cost.** Output tokens cost ~5x input tokens because each one is another
|
||||||
|
decode pass.
|
||||||
|
3. **Type errors and hallucination.** No matter how smart the model, the output
|
||||||
|
space is "any string", so it can emit an unexpected value, malformed JSON,
|
||||||
|
or a made-up fact. When the consumer is software — not a human reading it —
|
||||||
|
that is a deal-breaker, especially deep in a dependency chain.
|
||||||
|
|
||||||
|
TypeSafe's thesis: most *machine-consumed* intelligence is not "write me an
|
||||||
|
essay". It is small judgments — which bucket, how severe, is this true — that
|
||||||
|
software could consume directly if the model were shaped for it. Kahneman's
|
||||||
|
System 1 / System 2 distinction is the branding: Jev is fast intuition, LLMs
|
||||||
|
are slow deliberation.
|
||||||
|
|
||||||
|
## The interface
|
||||||
|
|
||||||
|
```
|
||||||
|
state + typed questions → typed probabilistic answers
|
||||||
|
```
|
||||||
|
|
||||||
|
One endpoint (`POST https://api.typesafe.ai/v1/systemone`), SDKs in Python
|
||||||
|
(`typesafe-sdk`) and TypeScript (`@typesafe-ai/sdk`). `state` is a string, JSON
|
||||||
|
object, or array. Limits: 64k tokens for state + all questions, text only (no
|
||||||
|
images).
|
||||||
|
|
||||||
|
### Example
|
||||||
|
|
||||||
|
```python
|
||||||
|
from typesafe_sdk import Choice, Noul, Score, TypeSafeClient
|
||||||
|
|
||||||
|
client = TypeSafeClient()
|
||||||
|
|
||||||
|
response = client.system_one(
|
||||||
|
state={
|
||||||
|
"ticket": {"subject": "Duplicate charge",
|
||||||
|
"messages": [{"from": "customer",
|
||||||
|
"text": "I was charged twice for order A-104."}]},
|
||||||
|
"refund_policy": "Duplicate charges are eligible for a refund.",
|
||||||
|
},
|
||||||
|
questions={
|
||||||
|
"department": Choice(instructions="Which team should handle this",
|
||||||
|
criteria={"billing": "Payment or subscription issues",
|
||||||
|
"technical": "Bugs or integration problems",
|
||||||
|
"sales": "Pricing or account questions"}),
|
||||||
|
"frustration": Score(instructions="How frustrated the customer appears",
|
||||||
|
criteria=["Calm, just stating facts",
|
||||||
|
"Frustrated but civil",
|
||||||
|
"Very angry, strong language"]),
|
||||||
|
"refund_wanted": Noul(instructions="The customer explicitly asks for a refund"),
|
||||||
|
"policy_supports": Noul(instructions="The stated refund policy covers this"),
|
||||||
|
},
|
||||||
|
)
|
||||||
|
print(response.answers["department"].choice, response.answers["department"].confidence)
|
||||||
|
```
|
||||||
|
|
||||||
|
One call returns every answer in 70-500 ms.
|
||||||
|
|
||||||
|
## The three primitives
|
||||||
|
|
||||||
|
| Primitive | Question it answers | Returns |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| **Noul** | "Is this true?" | a 0-1 probability. No separate confidence field — the number *is* the belief. |
|
||||||
|
| **Choice** | "Which of these?" | winning option + probability for every option (up to 255) + confidence |
|
||||||
|
| **Score** | "Where on this scale?" | a value that can land *between* levels (e.g. 1.35) + per-level probabilities + confidence |
|
||||||
|
|
||||||
|
Design rules from the docs, worth internalizing:
|
||||||
|
|
||||||
|
- Define Choice criteria as concrete situations, and always include an `other`
|
||||||
|
option so the model can say "nothing fits" instead of picking the closest
|
||||||
|
wrong thing.
|
||||||
|
- Score levels should be concrete situations ("service down, no workaround"),
|
||||||
|
not vague adjectives ("high").
|
||||||
|
- Questions in one call are evaluated **independently** against the same state.
|
||||||
|
A tenth question costs a few tokens and almost no latency, so the winning
|
||||||
|
pattern is *speculative fan-out*: ask everything, let code ignore what does
|
||||||
|
not apply. If question B truly depends on question A's answer, make a second
|
||||||
|
call with updated state.
|
||||||
|
|
||||||
|
## How it works
|
||||||
|
|
||||||
|
### Why one forward pass is possible
|
||||||
|
|
||||||
|
An LLM must decode sequentially because its answer space is effectively
|
||||||
|
unbounded (any string). Jev's answer space is **fixed and enumerated in
|
||||||
|
advance** — 255 options, a scale, or a boolean. A bounded answer space can be
|
||||||
|
*scored* rather than generated: one forward pass computes a probability for
|
||||||
|
every possible answer, no decode loop afterward.
|
||||||
|
|
||||||
|
That one fact explains three marketing claims at once:
|
||||||
|
|
||||||
|
- **Speed** (70-500 ms): one pass instead of hundreds. This is different from
|
||||||
|
Transformer training parallelism — LLM *inference* was always sequential,
|
||||||
|
and Jev parallelizes inference-time decision computation for bounded
|
||||||
|
outputs only.
|
||||||
|
- **Free output tokens** ($0.042/MTok input, $0 output): there is nothing
|
||||||
|
after the single pass to meter.
|
||||||
|
- **"Cannot hallucinate"**: outputs are members of a predefined set by
|
||||||
|
construction; a malformed string was never a possible output. Careful: this
|
||||||
|
means *no schema errors*, not *always correct*. Jev can confidently pick the
|
||||||
|
wrong category. "Calibration" and "correctness" are different properties.
|
||||||
|
|
||||||
|
What the architecture actually is — encoder-only transformer with
|
||||||
|
classification heads, diffusion variant, something purpose-built — is
|
||||||
|
**undisclosed**. Treat all such explanations as informed speculation.
|
||||||
|
|
||||||
|
### RLCD: training for honesty about uncertainty
|
||||||
|
|
||||||
|
TypeSafe's training method, RLCD (Reinforcement Learning for Calibrated
|
||||||
|
Decisions), sits beside its two better-known cousins:
|
||||||
|
|
||||||
|
| Method | Reward signal | Used for |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| RLHF | human raters prefer this output | chat models (the lineage Jev's founder, Diogo Almeida, worked on at OpenAI) |
|
||||||
|
| RLVR | output passes a programmatic check | math/code reasoning |
|
||||||
|
| **RLCD** | stated confidence matches actual accuracy | Jev |
|
||||||
|
|
||||||
|
Calibration, concretely: a weather forecaster who says "70% chance of rain" on
|
||||||
|
100 days should get rain on roughly 70 of them. RLCD rewards the *match*
|
||||||
|
between confidence and outcomes, not persuasiveness. The payoff for software:
|
||||||
|
you can branch on the number. Act automatically at 0.95, ask a human at 0.55 —
|
||||||
|
but only if the number is honest *in aggregate*.
|
||||||
|
|
||||||
|
Two honest caveats. First, calibration is a distribution property: an
|
||||||
|
individual 0.9 answer can still be wrong. Second, TypeSafe's accuracy claims
|
||||||
|
come from their own "workflow evals" (agreement with the average of two
|
||||||
|
frontier LLMs), not from a public reproducible benchmark. The latency/cost
|
||||||
|
mechanism stands on its own logic; the accuracy numbers are so far self-run.
|
||||||
|
|
||||||
|
## Worked examples
|
||||||
|
|
||||||
|
**1. Support-ticket triage (the canonical case).** The code example above:
|
||||||
|
category, frustration, refund intent, and policy coverage answered in one
|
||||||
|
call; Python branches on the answers. On a million tickets, the launch-week
|
||||||
|
arithmetic gives ~$6.5k instead of ~$30k with most tickets answered in under
|
||||||
|
half a second.
|
||||||
|
|
||||||
|
**2. Code review risk matrix** (`devagrawal09/jev-review`). A staged reviewer:
|
||||||
|
first a Noul risk matrix over a diff, then Choice/Score profiles per file, then
|
||||||
|
severity. Cheap filters decide which files a big model looks at.
|
||||||
|
|
||||||
|
**3. Browser agent** (`browser-use/jev-ultrafast`). Each page becomes a
|
||||||
|
numbered element table; one Jev call picks both the operation (CLICK / TYPE /
|
||||||
|
SCROLL / DONE / BLOCKED) and its target; a small LLM runs only to type text.
|
||||||
|
Booked a flight on real Google Flights in 7.1 s for $0.0039.
|
||||||
|
|
||||||
|
**4. Playing Doom.** Game state as a structured object in, action Choice out,
|
||||||
|
~10 decisions/second. Games are the clean demo of *judgment per tick* — the
|
||||||
|
same shape as a trading bot deciding per 300 ms block or a drone's ~2.5 Hz
|
||||||
|
tactical layer sitting above a code-owned flight controller.
|
||||||
|
|
||||||
|
**5. Economics at scale** (1kpapers.com). 1,018 papers summarized by a
|
||||||
|
generative model ($3.99), classified into 24 topics by Jev — **$0.08**, median
|
||||||
|
256 ms per paper. Generative model writes, decision model classifies.
|
||||||
|
|
||||||
|
## What Jev is bad at (per TypeSafe's own "jaggedness" page)
|
||||||
|
|
||||||
|
- **Reads literally.** It answers the question you wrote, not the one you
|
||||||
|
meant. Negations and implied conditions land at face value.
|
||||||
|
- **Not a calculator.** Counting, date arithmetic, anything orderable-as-text
|
||||||
|
is unreliable. Iterate in code, ask one Noul per item.
|
||||||
|
- **Context rot.** Accuracy drops when state is padded with material the
|
||||||
|
question does not need. Retrieve and filter first.
|
||||||
|
- **State is not hostile.** Text engineered to argue for its own classification
|
||||||
|
can move the answer. User-controlled content in state is a threat model you
|
||||||
|
must handle.
|
||||||
|
- **Does not generate.** No strings, no code, no summaries. If you need a
|
||||||
|
value extracted from free text, get candidates with code or a generative
|
||||||
|
model, then let Jev pick.
|
||||||
|
- It also reads *literally*: contradictory criteria confuse it.
|
||||||
|
|
||||||
|
## Status of the claims
|
||||||
|
|
||||||
|
| Claim | Status |
|
||||||
|
| --- | --- |
|
||||||
|
| Three primitives, one parallel pass, RLCD by name | confirmed (TypeSafe's own materials) |
|
||||||
|
| RLCD loss/reward construction, architecture | undisclosed |
|
||||||
|
| $0.042/MTok, 70-500 ms, 20-200x / 40-400x gains | self-reported, not independently verified |
|
||||||
|
| Workflow-eval accuracy vs GPT-6 Astra / Fable 5.1 average | internal methodology, no public ground truth; one chart showed Jev below Sonnet 5 raw accuracy |
|
||||||
|
| "Zero hallucinations" | = zero schema/type errors by construction; semantic errors still possible |
|
||||||
|
|
||||||
|
## Sources
|
||||||
|
|
||||||
|
- TypeSafe launch post: <https://typesafe.ai/blog/introducing-system-one-models-and-jev>
|
||||||
|
- Docs: <https://docs.typesafe.ai/>
|
||||||
|
- Mechanism explainer (RLCD, single-pass, confirmed-vs-speculative): <https://www.explainx.ai/blog/how-does-jev-work-rlcd-system-one-model-explained-2026>
|
||||||
|
- Practical guide with shipped demos and failure modes: <https://dev.to/valyuai/how-to-use-jev-a-practical-guide-to-typesafes-system-one-model-g5e>
|
||||||
|
- The Register's critical writeup: <https://www.theregister.com/ai-and-ml/2026/09/16/typesafe-ai-debuts-model-for-machines-that-plays-doom/5296711>
|
||||||
|
- Project reference gist: <https://gist.github.com/pjburnhill/adf8d28efcad9df037bfdece178ef965>
|
||||||
120
docs/02-jev-in-the-pi-harness.md
Normal file
120
docs/02-jev-in-the-pi-harness.md
Normal file
@ -0,0 +1,120 @@
|
|||||||
|
# Jev in the pi (Oh My Pi) harness
|
||||||
|
|
||||||
|
Use cases for a decision model inside a coding-agent harness. Every claim below
|
||||||
|
is grounded in the harness's own docs (`omp://`); Jev facts come from
|
||||||
|
[01-what-is-jev.md](./01-what-is-jev.md). The mental filter for each idea: Jev
|
||||||
|
answers *bounded* questions ("is this X?", "which bucket?", "how severe?") in
|
||||||
|
~100-500 ms for ~$0.0002. The harness's LLM does the thinking and the writing.
|
||||||
|
|
||||||
|
## Where the harness can host a decision model
|
||||||
|
|
||||||
|
| Surface | Mechanism | File location |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| Hooks / extensions | event bus with `tool_call`, `tool_result`, `context`, `before_agent_start`, `session_*` handlers | `<cwd>/.omp/hooks/pre/*.ts`, `~/.omp/agent/hooks/pre/*.ts` |
|
||||||
|
| Custom tools | model-callable functions, Zod/arktype schemas | `~/.omp/agent/tools`, `.omp/tools` |
|
||||||
|
| Skills | `SKILL.md` capability packs | `~/.omp/agent/skills/<name>/` (also `managed-skills` via `autolearn.enabled`) |
|
||||||
|
| Context files | `AGENTS.md` (user + project), `RULES.md` (sticky) | `~/.omp/agent/AGENTS.md`, `.omp/AGENTS.md` |
|
||||||
|
| System prompt | `SYSTEM.md` (replace), `APPEND_SYSTEM.md` (add), `PERSONALITY.md` | `~/.omp/agent/...` |
|
||||||
|
| Memory pipeline | two-phase extraction+consolidation over past sessions | `memory.backend: local`, phases run on `default` / `smol` model roles |
|
||||||
|
| Raw material | persisted sessions (JSONL), prompt history (SQLite FTS5) | `~/.omp/agent/sessions/.../*.jsonl`, `~/.omp/agent/history.db` |
|
||||||
|
| Model routing | `models.yml`, roles (`smol`, `tiny`, `task`, `advisor`, ...) | `~/.omp/agent/models.yml` |
|
||||||
|
|
||||||
|
Jev is **not** a model provider in `models.yml` — it cannot chat, stream, or
|
||||||
|
hold a conversation, so it cannot be a session model. It plugs in through
|
||||||
|
hooks, custom tools, and batch jobs (see
|
||||||
|
[03-dreaming-with-jev.md](./03-dreaming-with-jev.md)).
|
||||||
|
|
||||||
|
## Use cases, ranked by fit
|
||||||
|
|
||||||
|
### 1. Pre-flight tool guardrail (`tool_call` hook)
|
||||||
|
|
||||||
|
The documented hook example blocks `bash` commands containing `rm -rf` by
|
||||||
|
string match. Jev upgrades this to a semantic risk gate: state = the command +
|
||||||
|
cwd context; questions = Noul "destroys or exfiltrates data", Score "blast
|
||||||
|
radius", Choice over {allow, confirm, block}. Confidence-gate it: auto-allow
|
||||||
|
when safe > 0.9, ask the user in between, hard-block above a destruction
|
||||||
|
threshold. Same shape applies to `write`/`edit` paths (protected files) and to
|
||||||
|
`bash` chains that look innocent but end in a push.
|
||||||
|
|
||||||
|
Caveat: the model reads literally; keep the deterministic regex checks too —
|
||||||
|
Jev is the fuzzy layer on top, not the replacement.
|
||||||
|
|
||||||
|
### 2. Prompt-injection screen on tool results (`tool_result` hook)
|
||||||
|
|
||||||
|
Web pages, issue bodies, and arbitrary repo files can contain text engineered
|
||||||
|
to steer the agent. A `tool_result` handler can ask Jev: Noul "contains
|
||||||
|
instructions attempting to steer the agent or exfiltrate secrets", and if it
|
||||||
|
fires, annotate or redact the content before it enters context. Cost per
|
||||||
|
screening ~$0.0002 — cheap enough to run on every external read. Note the
|
||||||
|
docs' own warning cuts both ways: injection-shaped state can also move Jev's
|
||||||
|
answer, so treat the verdict as a tripwire for human-visible annotation, not
|
||||||
|
an autonomous veto.
|
||||||
|
|
||||||
|
### 3. Verification of agent output (the "verify everything" pattern)
|
||||||
|
|
||||||
|
An `agent_end` hook or a `verify` custom tool that scores the turn's outcome:
|
||||||
|
Noul per acceptance criterion ("the reported fix actually reproduces-and-tests
|
||||||
|
the changed path"), Score "evidence quality", Choice over
|
||||||
|
{complete, partial, off-track}. This is the System One workflow shape TypeSafe
|
||||||
|
built its evals around, and it plugs directly into the harness's verify-before-
|
||||||
|
yield discipline: an independent cheap judge over the transcript before the
|
||||||
|
turn is declared done.
|
||||||
|
|
||||||
|
### 4. Memory pipeline pre-filter (cheap Phase 1)
|
||||||
|
|
||||||
|
The local memory pipeline runs per-session extraction on the `default`-role
|
||||||
|
model for every changed session (cap: 64 sessions/startup, 4k-token inputs).
|
||||||
|
A Jev pre-pass over each session could ask: Noul "contains durable signal"
|
||||||
|
(decisions, constraints, resolved failures), Score "density of signal",
|
||||||
|
Noul "worth burning an LLM extraction on" — so the expensive extractor only
|
||||||
|
runs on sessions Jev rates. At 64 sessions x ~4k tokens the LLM pass is a few
|
||||||
|
cents; Jev's version is ~1% of that and ~100x faster, which is what makes
|
||||||
|
scanning *every* session every cycle affordable instead of capped.
|
||||||
|
|
||||||
|
### 5. Task routing / cascade (which agent, which model)
|
||||||
|
|
||||||
|
The `task` tool takes an `agent` choice (scout / task / reviewer /
|
||||||
|
security-reviewer) and `workpool` spreads items. A Jev front-end classifies
|
||||||
|
each item: Choice over agent types, Score "difficulty", Noul "read-only
|
||||||
|
research" — routing cheap items to scout, reserving expensive models. Same
|
||||||
|
cascade logic TypeSafe demos on support tickets, applied to subagent spawn.
|
||||||
|
This is advisory: a wrapper (custom tool or hook around `task`) makes the
|
||||||
|
call; the harness still owns control flow.
|
||||||
|
|
||||||
|
### 6. Config-drift guard on self-edits
|
||||||
|
|
||||||
|
When an agent writes to `AGENTS.md`, `RULES.md`, or a skill (the dreaming
|
||||||
|
pipeline in doc 03 does exactly this), a hook can gate the write: Noul "this
|
||||||
|
change encodes a one-off incident as a permanent rule", Noul "this rule
|
||||||
|
contradicts an existing rule in state". Cheap insurance against an agent
|
||||||
|
slowly cluttering its own configuration — the failure mode auto-learn systems
|
||||||
|
drift into.
|
||||||
|
|
||||||
|
### 7. Batch triage over history: the RSI dreaming loop
|
||||||
|
|
||||||
|
The biggest one gets its own document: replaying past sessions to mine
|
||||||
|
improvement opportunities (new skills, AGENTS.md/RULES.md edits, system-prompt
|
||||||
|
updates, new tools). Jev is what makes scanning *all* past interactions
|
||||||
|
economical. See [03-dreaming-with-jev.md](./03-dreaming-with-jev.md).
|
||||||
|
|
||||||
|
## Not Jev-shaped (do not force it)
|
||||||
|
|
||||||
|
- **Session titles, commit messages, summaries, compaction** — these are
|
||||||
|
*string generation*; the harness routes them to `tiny`/`smol` LLM roles for
|
||||||
|
good reason. Jev has no decoder; it cannot write them.
|
||||||
|
- **Code generation, refactors, reviews that need new text** — bounded-output
|
||||||
|
scoring can *rank* candidates but cannot produce them.
|
||||||
|
- **Anything needing knowledge outside the state you send** — Jev has no
|
||||||
|
world knowledge beyond state; no lookups. Retrieve first, judge second.
|
||||||
|
- **Image/screens understanding** — text-only state (as of Jev 1.13).
|
||||||
|
|
||||||
|
## Honest limits before betting on any of this
|
||||||
|
|
||||||
|
- Early access product; `jev-latest` currently resolves to `jev-1.13.0` and
|
||||||
|
moving GPU capacity means rate limits (250k tok/s, 1200 req/min) and
|
||||||
|
behavior can shift. Pin the version and log the response's `model` field.
|
||||||
|
- All performance numbers are TypeSafe's own, launch-week, unreproduced.
|
||||||
|
- Calibration ≠ correctness: validate every threshold against local outcomes
|
||||||
|
before letting it act autonomously.
|
||||||
|
- State ceiling: 64k tokens per call. Whole sessions don't fit; segment first
|
||||||
|
(doc 03 does).
|
||||||
246
docs/03-dreaming-with-jev.md
Normal file
246
docs/03-dreaming-with-jev.md
Normal file
@ -0,0 +1,246 @@
|
|||||||
|
# Dreaming with Jev: a self-improvement method for the pi configuration
|
||||||
|
|
||||||
|
*RSI project — proposal. Status: design, ready for staged implementation.*
|
||||||
|
|
||||||
|
## What "dreaming" is
|
||||||
|
|
||||||
|
Dreaming is a scheduled batch job that replays the agent's past interactions
|
||||||
|
and mines them for improvement opportunities, then turns the best ones into
|
||||||
|
configuration changes:
|
||||||
|
|
||||||
|
- new or updated **skills** (`~/.omp/agent/skills/<name>/SKILL.md`)
|
||||||
|
- **context file** edits (`~/.omp/agent/AGENTS.md`, `RULES.md`)
|
||||||
|
- **system prompt** updates (`APPEND_SYSTEM.md`, `PERSONALITY.md`)
|
||||||
|
- new **custom tools** (`~/.omp/agent/tools/*.ts`)
|
||||||
|
|
||||||
|
Nothing lands directly. Every output is a reviewable artifact — a Gitea PR
|
||||||
|
against the config repo and/or a pm-board proposal — and the user merges.
|
||||||
|
|
||||||
|
## Why Jev, and why now
|
||||||
|
|
||||||
|
The harness already has the raw material: every persisted session is a JSONL
|
||||||
|
file under `~/.omp/agent/sessions/<encoded-cwd>/`, plus prompt history in
|
||||||
|
`~/.omp/agent/history.db` (FTS5). The local memory pipeline (`memory.backend:
|
||||||
|
local`) already replays changed sessions — but it runs a `default`-role LLM per
|
||||||
|
session, caps at 64 sessions per startup, 300 scanned, with 12-hour idle
|
||||||
|
gating. It is an *understanding* pipeline, not an *improvement* pipeline.
|
||||||
|
|
||||||
|
The bottleneck is triage: deciding which parts of which sessions are worth a
|
||||||
|
$0.01+ LLM pass. That is a bounded judgment — "is there a durable improvement
|
||||||
|
opportunity in this segment, and what kind?" — exactly Jev's shape:
|
||||||
|
|
||||||
|
- input-only billing at $0.042/MTok, free output
|
||||||
|
- 70-500 ms per call, questions in one call evaluated in parallel
|
||||||
|
- calibrated probabilities → thresholds you can tune against outcomes
|
||||||
|
- every candidate question is a Noul/Score/Choice, no string parsing
|
||||||
|
|
||||||
|
At those economics the dreamer can screen **everything** (all sessions, all
|
||||||
|
segments) every night and escalate ~1-2% to an LLM. That inverts the usual
|
||||||
|
sampling-with-heuristics design: no session is skipped because it was cheap to
|
||||||
|
look.
|
||||||
|
|
||||||
|
## Pipeline
|
||||||
|
|
||||||
|
```
|
||||||
|
S0 harvest → S1 segment → S2 jev triage → S3 cluster → S4 LLM synthesis
|
||||||
|
↓
|
||||||
|
S6 calibration ← S5 human gate (PR + pm-board proposals)
|
||||||
|
```
|
||||||
|
|
||||||
|
### S0 — Harvest
|
||||||
|
|
||||||
|
Sources:
|
||||||
|
|
||||||
|
- `~/.omp/agent/sessions/<bucket>/*.jsonl` — parse per the session format:
|
||||||
|
fixed-width 256-byte title slot first, then session header, then entries.
|
||||||
|
Keep `message` entries (roles are camelCase: `user`, `assistant`,
|
||||||
|
`toolResult`); note `toolCall` blocks live inside assistant content.
|
||||||
|
Skip or downweight: `compaction`, `branch_summary`, `custom` (opaque),
|
||||||
|
entries after `reset_boundary`.
|
||||||
|
- `~/.omp/agent/history.db` (`history` table) — bare user prompts are a cheap
|
||||||
|
signal of repeated requests.
|
||||||
|
- pm-board `.pm/` state — accepted/rejected proposals tell you which past
|
||||||
|
suggestions the user actually wanted.
|
||||||
|
- Generated skills and `learned.md` under the memory root — what has already
|
||||||
|
been learned, so the dreamer proposes deltas, not duplicates.
|
||||||
|
|
||||||
|
### S1 — Segment
|
||||||
|
|
||||||
|
Cut each session into units small enough for one Jev call (state ceiling 64k
|
||||||
|
tokens; aim ≤4k). Natural cuts: one user intent → the assistant trajectory
|
||||||
|
that resolved it (tool calls + results + verdicts). Attach metadata: project
|
||||||
|
cwd, date, exit reason (`session_exit` entries), todo outcomes.
|
||||||
|
|
||||||
|
### S2 — Jev triage (the speculative fan-out)
|
||||||
|
|
||||||
|
One call per segment; all questions ride the same call:
|
||||||
|
|
||||||
|
```python
|
||||||
|
response = jev.system_one(
|
||||||
|
state={
|
||||||
|
"segment": seg.text, # redacted (see Safety)
|
||||||
|
"meta": {"cwd": seg.cwd, "date": seg.date, "exit": seg.exit_reason},
|
||||||
|
},
|
||||||
|
questions={
|
||||||
|
"friction": Noul("The user had to correct, repeat, or unblock the agent"),
|
||||||
|
"repeated": Noul("This is a routine workflow, done by hand, that recurs"),
|
||||||
|
"failure": Noul("The attempt failed or needed several retries"),
|
||||||
|
"config_gap": Noul("The failure or friction traces to a missing skill, rule, prompt guidance, or tool"),
|
||||||
|
"waste": Noul("The path taken was clearly inefficient vs a simpler one"),
|
||||||
|
"already_configured": Noul("The behavior that would fix this is already written in the provided current-config summary"),
|
||||||
|
"value": Score("How much lasting improvement is available here",
|
||||||
|
criteria=["Nothing actionable",
|
||||||
|
"Cosmetic preference",
|
||||||
|
"Workflow would improve one class of tasks",
|
||||||
|
"Recurring failure class would be prevented",
|
||||||
|
"Systemic issue affecting many future sessions"]),
|
||||||
|
"opportunity": Choice("What kind of change would fix it",
|
||||||
|
criteria={"new_skill": "A repeatable workflow deserves its own SKILL.md",
|
||||||
|
"rule_update": "A hard rule (RULES.md) or context guidance (AGENTS.md) is missing or wrong",
|
||||||
|
"prompt_update": "System-prompt behavior guidance should change (APPEND_SYSTEM.md)",
|
||||||
|
"new_tool": "A deterministic helper tool is missing",
|
||||||
|
"memory_only": "Worth remembering, not config-worthy",
|
||||||
|
"nothing": "No improvement available"}),
|
||||||
|
},
|
||||||
|
)
|
||||||
|
```
|
||||||
|
|
||||||
|
Pass a compact summary of the *current* config (skill names + descriptions,
|
||||||
|
rules titles, APPEND_SYSTEM.md) in state so `already_configured` can veto
|
||||||
|
duplicate proposals — this is the "context rot" safeguard: send only the fields
|
||||||
|
each question needs.
|
||||||
|
|
||||||
|
**Gating:** proceed only where `config_right_side` confidence is real:
|
||||||
|
|
||||||
|
```python
|
||||||
|
keep = (a["config_gap"].noul > 0.7 and a["already_configured"].noul < 0.3
|
||||||
|
and a["value"].score >= 2.0 and a["value"].confidence > 0.6)
|
||||||
|
```
|
||||||
|
|
||||||
|
Flat distributions ("value" confidence < 0.5) are a signal the criteria are
|
||||||
|
wrong for that segment — log them, do not guess. Thresholds live in one config
|
||||||
|
file; the calibration loop (S6) tunes them.
|
||||||
|
|
||||||
|
### S3 — Cluster and dedupe
|
||||||
|
|
||||||
|
Ten segments about git-push friction are one proposal, not ten. Group by
|
||||||
|
`opportunity_type`, then similarity over segment embeddings (or a second,
|
||||||
|
tiny Jev Choice over candidate groups — still bounded, still cheap). Keep the
|
||||||
|
top cluster exemplars (highest `value` score), cap proposals per dream run
|
||||||
|
(default 5) to bound blast radius.
|
||||||
|
|
||||||
|
### S4 — Synthesis (this is where the LLM earns its price)
|
||||||
|
|
||||||
|
Jev cannot write. Hand each surviving cluster — evidence links (session ids,
|
||||||
|
timestamps, the exact friction quotes) plus the relevant current-config files —
|
||||||
|
to a harness run (`omp -p --no-session`, the same pattern the pm-address timer
|
||||||
|
uses to reconcile boards). The LLM drafts one artifact per cluster:
|
||||||
|
|
||||||
|
| Cluster verdict | Artifact it drafts |
|
||||||
|
| --- | --- |
|
||||||
|
| `new_skill` | `<name>/SKILL.md` with `name` + `description` frontmatter (description required for native discovery), body as a workflow with evidence-linked examples |
|
||||||
|
| `rule_update` | diff for `~/.omp/agent/RULES.md` (hard rules only — short) or `AGENTS.md` (durable background) |
|
||||||
|
| `prompt_update` | diff for `APPEND_SYSTEM.md` |
|
||||||
|
| `new_tool` | `.omp/tools/<name>.ts` custom-tool module sketch (factory + Zod schema), marked needs-implementation |
|
||||||
|
|
||||||
|
### S5 — Human gate
|
||||||
|
|
||||||
|
Artifacts are written to a dream workspace branch:
|
||||||
|
|
||||||
|
- one Gitea PR per dream run against the config repo (per the standing branch
|
||||||
|
workflow: never commit to main; the user merges), and
|
||||||
|
- pm-board proposals in `.pm/proposals/*.md` (frontmatter `op:`/`id:`/
|
||||||
|
`section:`/`title:`) for board-tracked items.
|
||||||
|
|
||||||
|
The user's accept/reject is the ground-truth label for S6.
|
||||||
|
|
||||||
|
### S6 — Calibration loop
|
||||||
|
|
||||||
|
Log every verdict with its probability and its eventual outcome:
|
||||||
|
|
||||||
|
```json
|
||||||
|
{"segment": "...", "config_gap_noul": 0.83, "value_score": 2.4,
|
||||||
|
"cluster": "git-push-friction", "outcome": "accepted", "artifact": "PR #12"}
|
||||||
|
```
|
||||||
|
|
||||||
|
Then measure, per question, whether the 0.8 bucket really lands ~80% accepted.
|
||||||
|
This is RLCD's philosophy applied to the dreamer itself: confidence must track
|
||||||
|
outcomes, and drift in that curve is the alarm that thresholds (or the criteria
|
||||||
|
wording — Jev reads literally) need re-tuning. Calibration data is also what
|
||||||
|
justifies loosening the human gate later, if ever.
|
||||||
|
|
||||||
|
## Wiring into this harness
|
||||||
|
|
||||||
|
- **Scheduler:** mirror the pm-address pattern (hourly `omp -p --no-session`
|
||||||
|
reconciliation in `deploy/README.md`) — a nightly `dream` timer/cron that
|
||||||
|
invokes the pipeline, then a synthesis `omp` run over the shortlist.
|
||||||
|
- **Secrets:** the memory pipeline redacts secret patterns before writing
|
||||||
|
memory artifacts; the dreamer must do the same *before* anything leaves the
|
||||||
|
machine as Jev state, and re-redact before synthesis.
|
||||||
|
- **Overlap with memory:** memory consolidation produces knowledge summaries;
|
||||||
|
dreaming produces config changes. They share S0. If `memory.backend: local`
|
||||||
|
is enabled, reuse its per-session extraction cadence markers instead of
|
||||||
|
re-parsing unchanged sessions (the pipeline tracks processed-since mtime).
|
||||||
|
- **Where outputs land:** skills into `~/.omp/agent/skills/<name>/` (native
|
||||||
|
provider, priority 100; beats same-named lower providers), rules per the
|
||||||
|
RULES.md contract (short, sticky, native locations only), tools into
|
||||||
|
`~/.omp/agent/tools/`.
|
||||||
|
|
||||||
|
## Cost model
|
||||||
|
|
||||||
|
Per dream run (rough, at self-reported prices):
|
||||||
|
|
||||||
|
| Stage | Volume | Cost |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| S2 Jev: 300 sessions → ~1,200 segments x ~4k tokens | ~5 MTok input | **~$0.21** |
|
||||||
|
| S4 LLM synthesis: ~5 clusters x ~15k tokens | ~75k tok | cents–$1 |
|
||||||
|
| S0-S3, S6 code + SQLite | — | negligible |
|
||||||
|
|
||||||
|
A full-history dream costs well under $1. The LLM synthesis spend is the
|
||||||
|
larger line item and scales with proposal count, which S3 caps.
|
||||||
|
|
||||||
|
## Safety: this is self-modification, treat it like it
|
||||||
|
|
||||||
|
1. **Human gate.** No direct writes to live config. PRs + proposals only; the
|
||||||
|
user merges. (Also required by the house rule: no commits to main.)
|
||||||
|
2. **Blast-radius cap.** ≤5 proposals per run, each with evidence links; a
|
||||||
|
proposal without a session citation is rejected by construction.
|
||||||
|
3. **Injection.** Sessions contain untrusted content (web pages, issue text)
|
||||||
|
and "state is not hostile" applies — content can argue for its own
|
||||||
|
improvement. Mitigations: redaction before Jev sees it, the LLM synthesizer
|
||||||
|
independently reviews evidence, and the human gate sees the evidence too.
|
||||||
|
4. **Feedback loops.** The dreamer must not grade its own artifacts as
|
||||||
|
"accepted" — outcomes come from the human's merge/reject actions only.
|
||||||
|
5. **Reversibility.** Config lives in git; every accepted proposal is a
|
||||||
|
reviewable commit; `RULES.md` stays short so the sticky layer can't bloat.
|
||||||
|
|
||||||
|
## Implementation notes
|
||||||
|
|
||||||
|
- Adapter interface first: `JevClient` with three backends — `jev` (real API,
|
||||||
|
needs `TYPESAFE_API_KEY` early access, or Vercel AI gateway), `llm` (a
|
||||||
|
frontier LLM constrained to the same schema, e.g. via TypeSafe's published
|
||||||
|
`system-one-adapter-python` shim), and `mock` (deterministic, for tests).
|
||||||
|
The interface is the design; the backend is swappable, so the pipeline can
|
||||||
|
be built and audited before the key exists.
|
||||||
|
- Pin `jev-1.13.0` in config, log the `model` field from every response.
|
||||||
|
- Dry-run mode writes an audit report (what would have been proposed, with
|
||||||
|
Jev's raw probabilities) and nothing else.
|
||||||
|
|
||||||
|
## Rollout
|
||||||
|
|
||||||
|
| Milestone | Gate |
|
||||||
|
| --- | --- |
|
||||||
|
| M0: parser + S2 dry run over last 30 days, human reads the audit report | report is sensibly targeted (eyeball) |
|
||||||
|
| M1: full pipeline, thresholds strict (`config_gap > 0.8`), ≤3 proposals/run | user accepts ≥1 real artifact |
|
||||||
|
| M2: weekly cadence, thresholds tuned from calibration data | calibration curve within ±15pp of diagonal |
|
||||||
|
|
||||||
|
## Open questions
|
||||||
|
|
||||||
|
- Does Jev's calibration hold on *this* corpus (session transcripts are
|
||||||
|
long, messy, and self-authored)? M0 exists to answer this before anything
|
||||||
|
acts.
|
||||||
|
- Cluster-level evidence may need a second Jev pass with the current config in
|
||||||
|
state (serial call, justified dependency per the primitives' independence
|
||||||
|
rule).
|
||||||
|
- Whether S4 synthesis should also run the doc-02 "config-drift guard" on its
|
||||||
|
own drafts before proposing them.
|
||||||
Loading…
x
Reference in New Issue
Block a user