Compare commits
1 Commits
main
...
chore/pm-r
| Author | SHA1 | Date | |
|---|---|---|---|
|
5202bd98be |
7
.gitignore
vendored
7
.gitignore
vendored
@ -1,2 +1,9 @@
|
|||||||
__pycache__/
|
__pycache__/
|
||||||
*.pyc
|
*.pyc
|
||||||
|
|
||||||
|
# pm-board runner output and legacy runtime files (never tracked)
|
||||||
|
.pm/runs.jsonl
|
||||||
|
.pm/address-state.json
|
||||||
|
.pm/address-history.jsonl
|
||||||
|
.pm/feedback-state.json
|
||||||
|
.pm/outbox.md
|
||||||
|
|||||||
@ -1,3 +1,4 @@
|
|||||||
|
gitea_url: "https://git.jajahome.duckdns.org/omp/rsi"
|
||||||
requirements:
|
requirements:
|
||||||
- id: R1
|
- id: R1
|
||||||
title: Jev research summary
|
title: Jev research summary
|
||||||
|
|||||||
@ -1 +0,0 @@
|
|||||||
https://git.jajahome.duckdns.org/omp/rsi
|
|
||||||
@ -1,211 +0,0 @@
|
|||||||
# Jev: what it does and how it works
|
|
||||||
|
|
||||||
Research summary of TypeSafe AI's Jev, launched 15 September 2026. Written so an
|
|
||||||
undergraduate with one AI course behind them can follow it. Sources at the
|
|
||||||
bottom; claims are tagged confirmed, self-reported, or speculative.
|
|
||||||
|
|
||||||
## TL;DR
|
|
||||||
|
|
||||||
Jev is not a chatbot and not a smaller LLM. It is a **decision model**: you send
|
|
||||||
it a piece of state (text or JSON) plus a set of pre-defined questions, and it
|
|
||||||
returns typed answers — a category, a number on a scale, or a probability — each
|
|
||||||
with calibrated probabilities. It never generates text. That single design
|
|
||||||
choice makes it ~100x faster and ~100-400x cheaper than a frontier LLM on these
|
|
||||||
narrow questions, and makes malformed answers structurally impossible.
|
|
||||||
|
|
||||||
## The problem it solves
|
|
||||||
|
|
||||||
A normal LLM returns **strings**. To use a string inside software you must
|
|
||||||
parse it, validate it, and hope the model did not go off the rails. Three costs
|
|
||||||
follow:
|
|
||||||
|
|
||||||
1. **Latency.** LLMs generate one token at a time, each conditioned on the
|
|
||||||
last. A 500-token answer is ~500 sequential forward passes. Frontier
|
|
||||||
responses take 3s to 329s end to end.
|
|
||||||
2. **Cost.** Output tokens cost ~5x input tokens because each one is another
|
|
||||||
decode pass.
|
|
||||||
3. **Type errors and hallucination.** No matter how smart the model, the output
|
|
||||||
space is "any string", so it can emit an unexpected value, malformed JSON,
|
|
||||||
or a made-up fact. When the consumer is software — not a human reading it —
|
|
||||||
that is a deal-breaker, especially deep in a dependency chain.
|
|
||||||
|
|
||||||
TypeSafe's thesis: most *machine-consumed* intelligence is not "write me an
|
|
||||||
essay". It is small judgments — which bucket, how severe, is this true — that
|
|
||||||
software could consume directly if the model were shaped for it. Kahneman's
|
|
||||||
System 1 / System 2 distinction is the branding: Jev is fast intuition, LLMs
|
|
||||||
are slow deliberation.
|
|
||||||
|
|
||||||
## The interface
|
|
||||||
|
|
||||||
```
|
|
||||||
state + typed questions → typed probabilistic answers
|
|
||||||
```
|
|
||||||
|
|
||||||
One endpoint (`POST https://api.typesafe.ai/v1/systemone`), SDKs in Python
|
|
||||||
(`typesafe-sdk`) and TypeScript (`@typesafe-ai/sdk`). `state` is a string, JSON
|
|
||||||
object, or array. Limits: 64k tokens for state + all questions, text only (no
|
|
||||||
images).
|
|
||||||
|
|
||||||
### Example
|
|
||||||
|
|
||||||
```python
|
|
||||||
from typesafe_sdk import Choice, Noul, Score, TypeSafeClient
|
|
||||||
|
|
||||||
client = TypeSafeClient()
|
|
||||||
|
|
||||||
response = client.system_one(
|
|
||||||
state={
|
|
||||||
"ticket": {"subject": "Duplicate charge",
|
|
||||||
"messages": [{"from": "customer",
|
|
||||||
"text": "I was charged twice for order A-104."}]},
|
|
||||||
"refund_policy": "Duplicate charges are eligible for a refund.",
|
|
||||||
},
|
|
||||||
questions={
|
|
||||||
"department": Choice(instructions="Which team should handle this",
|
|
||||||
criteria={"billing": "Payment or subscription issues",
|
|
||||||
"technical": "Bugs or integration problems",
|
|
||||||
"sales": "Pricing or account questions"}),
|
|
||||||
"frustration": Score(instructions="How frustrated the customer appears",
|
|
||||||
criteria=["Calm, just stating facts",
|
|
||||||
"Frustrated but civil",
|
|
||||||
"Very angry, strong language"]),
|
|
||||||
"refund_wanted": Noul(instructions="The customer explicitly asks for a refund"),
|
|
||||||
"policy_supports": Noul(instructions="The stated refund policy covers this"),
|
|
||||||
},
|
|
||||||
)
|
|
||||||
print(response.answers["department"].choice, response.answers["department"].confidence)
|
|
||||||
```
|
|
||||||
|
|
||||||
One call returns every answer in 70-500 ms.
|
|
||||||
|
|
||||||
## The three primitives
|
|
||||||
|
|
||||||
| Primitive | Question it answers | Returns |
|
|
||||||
| --- | --- | --- |
|
|
||||||
| **Noul** | "Is this true?" | a 0-1 probability. No separate confidence field — the number *is* the belief. |
|
|
||||||
| **Choice** | "Which of these?" | winning option + probability for every option (up to 255) + confidence |
|
|
||||||
| **Score** | "Where on this scale?" | a value that can land *between* levels (e.g. 1.35) + per-level probabilities + confidence |
|
|
||||||
|
|
||||||
Design rules from the docs, worth internalizing:
|
|
||||||
|
|
||||||
- Define Choice criteria as concrete situations, and always include an `other`
|
|
||||||
option so the model can say "nothing fits" instead of picking the closest
|
|
||||||
wrong thing.
|
|
||||||
- Score levels should be concrete situations ("service down, no workaround"),
|
|
||||||
not vague adjectives ("high").
|
|
||||||
- Questions in one call are evaluated **independently** against the same state.
|
|
||||||
A tenth question costs a few tokens and almost no latency, so the winning
|
|
||||||
pattern is *speculative fan-out*: ask everything, let code ignore what does
|
|
||||||
not apply. If question B truly depends on question A's answer, make a second
|
|
||||||
call with updated state.
|
|
||||||
|
|
||||||
## How it works
|
|
||||||
|
|
||||||
### Why one forward pass is possible
|
|
||||||
|
|
||||||
An LLM must decode sequentially because its answer space is effectively
|
|
||||||
unbounded (any string). Jev's answer space is **fixed and enumerated in
|
|
||||||
advance** — 255 options, a scale, or a boolean. A bounded answer space can be
|
|
||||||
*scored* rather than generated: one forward pass computes a probability for
|
|
||||||
every possible answer, no decode loop afterward.
|
|
||||||
|
|
||||||
That one fact explains three marketing claims at once:
|
|
||||||
|
|
||||||
- **Speed** (70-500 ms): one pass instead of hundreds. This is different from
|
|
||||||
Transformer training parallelism — LLM *inference* was always sequential,
|
|
||||||
and Jev parallelizes inference-time decision computation for bounded
|
|
||||||
outputs only.
|
|
||||||
- **Free output tokens** ($0.042/MTok input, $0 output): there is nothing
|
|
||||||
after the single pass to meter.
|
|
||||||
- **"Cannot hallucinate"**: outputs are members of a predefined set by
|
|
||||||
construction; a malformed string was never a possible output. Careful: this
|
|
||||||
means *no schema errors*, not *always correct*. Jev can confidently pick the
|
|
||||||
wrong category. "Calibration" and "correctness" are different properties.
|
|
||||||
|
|
||||||
What the architecture actually is — encoder-only transformer with
|
|
||||||
classification heads, diffusion variant, something purpose-built — is
|
|
||||||
**undisclosed**. Treat all such explanations as informed speculation.
|
|
||||||
|
|
||||||
### RLCD: training for honesty about uncertainty
|
|
||||||
|
|
||||||
TypeSafe's training method, RLCD (Reinforcement Learning for Calibrated
|
|
||||||
Decisions), sits beside its two better-known cousins:
|
|
||||||
|
|
||||||
| Method | Reward signal | Used for |
|
|
||||||
| --- | --- | --- |
|
|
||||||
| RLHF | human raters prefer this output | chat models (the lineage Jev's founder, Diogo Almeida, worked on at OpenAI) |
|
|
||||||
| RLVR | output passes a programmatic check | math/code reasoning |
|
|
||||||
| **RLCD** | stated confidence matches actual accuracy | Jev |
|
|
||||||
|
|
||||||
Calibration, concretely: a weather forecaster who says "70% chance of rain" on
|
|
||||||
100 days should get rain on roughly 70 of them. RLCD rewards the *match*
|
|
||||||
between confidence and outcomes, not persuasiveness. The payoff for software:
|
|
||||||
you can branch on the number. Act automatically at 0.95, ask a human at 0.55 —
|
|
||||||
but only if the number is honest *in aggregate*.
|
|
||||||
|
|
||||||
Two honest caveats. First, calibration is a distribution property: an
|
|
||||||
individual 0.9 answer can still be wrong. Second, TypeSafe's accuracy claims
|
|
||||||
come from their own "workflow evals" (agreement with the average of two
|
|
||||||
frontier LLMs), not from a public reproducible benchmark. The latency/cost
|
|
||||||
mechanism stands on its own logic; the accuracy numbers are so far self-run.
|
|
||||||
|
|
||||||
## Worked examples
|
|
||||||
|
|
||||||
**1. Support-ticket triage (the canonical case).** The code example above:
|
|
||||||
category, frustration, refund intent, and policy coverage answered in one
|
|
||||||
call; Python branches on the answers. On a million tickets, the launch-week
|
|
||||||
arithmetic gives ~$6.5k instead of ~$30k with most tickets answered in under
|
|
||||||
half a second.
|
|
||||||
|
|
||||||
**2. Code review risk matrix** (`devagrawal09/jev-review`). A staged reviewer:
|
|
||||||
first a Noul risk matrix over a diff, then Choice/Score profiles per file, then
|
|
||||||
severity. Cheap filters decide which files a big model looks at.
|
|
||||||
|
|
||||||
**3. Browser agent** (`browser-use/jev-ultrafast`). Each page becomes a
|
|
||||||
numbered element table; one Jev call picks both the operation (CLICK / TYPE /
|
|
||||||
SCROLL / DONE / BLOCKED) and its target; a small LLM runs only to type text.
|
|
||||||
Booked a flight on real Google Flights in 7.1 s for $0.0039.
|
|
||||||
|
|
||||||
**4. Playing Doom.** Game state as a structured object in, action Choice out,
|
|
||||||
~10 decisions/second. Games are the clean demo of *judgment per tick* — the
|
|
||||||
same shape as a trading bot deciding per 300 ms block or a drone's ~2.5 Hz
|
|
||||||
tactical layer sitting above a code-owned flight controller.
|
|
||||||
|
|
||||||
**5. Economics at scale** (1kpapers.com). 1,018 papers summarized by a
|
|
||||||
generative model ($3.99), classified into 24 topics by Jev — **$0.08**, median
|
|
||||||
256 ms per paper. Generative model writes, decision model classifies.
|
|
||||||
|
|
||||||
## What Jev is bad at (per TypeSafe's own "jaggedness" page)
|
|
||||||
|
|
||||||
- **Reads literally.** It answers the question you wrote, not the one you
|
|
||||||
meant. Negations and implied conditions land at face value.
|
|
||||||
- **Not a calculator.** Counting, date arithmetic, anything orderable-as-text
|
|
||||||
is unreliable. Iterate in code, ask one Noul per item.
|
|
||||||
- **Context rot.** Accuracy drops when state is padded with material the
|
|
||||||
question does not need. Retrieve and filter first.
|
|
||||||
- **State is not hostile.** Text engineered to argue for its own classification
|
|
||||||
can move the answer. User-controlled content in state is a threat model you
|
|
||||||
must handle.
|
|
||||||
- **Does not generate.** No strings, no code, no summaries. If you need a
|
|
||||||
value extracted from free text, get candidates with code or a generative
|
|
||||||
model, then let Jev pick.
|
|
||||||
- It also reads *literally*: contradictory criteria confuse it.
|
|
||||||
|
|
||||||
## Status of the claims
|
|
||||||
|
|
||||||
| Claim | Status |
|
|
||||||
| --- | --- |
|
|
||||||
| Three primitives, one parallel pass, RLCD by name | confirmed (TypeSafe's own materials) |
|
|
||||||
| RLCD loss/reward construction, architecture | undisclosed |
|
|
||||||
| $0.042/MTok, 70-500 ms, 20-200x / 40-400x gains | self-reported, not independently verified |
|
|
||||||
| Workflow-eval accuracy vs GPT-6 Astra / Fable 5.1 average | internal methodology, no public ground truth; one chart showed Jev below Sonnet 5 raw accuracy |
|
|
||||||
| "Zero hallucinations" | = zero schema/type errors by construction; semantic errors still possible |
|
|
||||||
|
|
||||||
## Sources
|
|
||||||
|
|
||||||
- TypeSafe launch post: <https://typesafe.ai/blog/introducing-system-one-models-and-jev>
|
|
||||||
- Docs: <https://docs.typesafe.ai/>
|
|
||||||
- Mechanism explainer (RLCD, single-pass, confirmed-vs-speculative): <https://www.explainx.ai/blog/how-does-jev-work-rlcd-system-one-model-explained-2026>
|
|
||||||
- Practical guide with shipped demos and failure modes: <https://dev.to/valyuai/how-to-use-jev-a-practical-guide-to-typesafes-system-one-model-g5e>
|
|
||||||
- The Register's critical writeup: <https://www.theregister.com/ai-and-ml/2026/09/16/typesafe-ai-debuts-model-for-machines-that-plays-doom/5296711>
|
|
||||||
- Project reference gist: <https://gist.github.com/pjburnhill/adf8d28efcad9df037bfdece178ef965>
|
|
||||||
@ -1,120 +0,0 @@
|
|||||||
# Jev in the pi (Oh My Pi) harness
|
|
||||||
|
|
||||||
Use cases for a decision model inside a coding-agent harness. Every claim below
|
|
||||||
is grounded in the harness's own docs (`omp://`); Jev facts come from
|
|
||||||
[01-what-is-jev.md](./01-what-is-jev.md). The mental filter for each idea: Jev
|
|
||||||
answers *bounded* questions ("is this X?", "which bucket?", "how severe?") in
|
|
||||||
~100-500 ms for ~$0.0002. The harness's LLM does the thinking and the writing.
|
|
||||||
|
|
||||||
## Where the harness can host a decision model
|
|
||||||
|
|
||||||
| Surface | Mechanism | File location |
|
|
||||||
| --- | --- | --- |
|
|
||||||
| Hooks / extensions | event bus with `tool_call`, `tool_result`, `context`, `before_agent_start`, `session_*` handlers | `<cwd>/.omp/hooks/pre/*.ts`, `~/.omp/agent/hooks/pre/*.ts` |
|
|
||||||
| Custom tools | model-callable functions, Zod/arktype schemas | `~/.omp/agent/tools`, `.omp/tools` |
|
|
||||||
| Skills | `SKILL.md` capability packs | `~/.omp/agent/skills/<name>/` (also `managed-skills` via `autolearn.enabled`) |
|
|
||||||
| Context files | `AGENTS.md` (user + project), `RULES.md` (sticky) | `~/.omp/agent/AGENTS.md`, `.omp/AGENTS.md` |
|
|
||||||
| System prompt | `SYSTEM.md` (replace), `APPEND_SYSTEM.md` (add), `PERSONALITY.md` | `~/.omp/agent/...` |
|
|
||||||
| Memory pipeline | two-phase extraction+consolidation over past sessions | `memory.backend: local`, phases run on `default` / `smol` model roles |
|
|
||||||
| Raw material | persisted sessions (JSONL), prompt history (SQLite FTS5) | `~/.omp/agent/sessions/.../*.jsonl`, `~/.omp/agent/history.db` |
|
|
||||||
| Model routing | `models.yml`, roles (`smol`, `tiny`, `task`, `advisor`, ...) | `~/.omp/agent/models.yml` |
|
|
||||||
|
|
||||||
Jev is **not** a model provider in `models.yml` — it cannot chat, stream, or
|
|
||||||
hold a conversation, so it cannot be a session model. It plugs in through
|
|
||||||
hooks, custom tools, and batch jobs (see
|
|
||||||
[03-dreaming-with-jev.md](./03-dreaming-with-jev.md)).
|
|
||||||
|
|
||||||
## Use cases, ranked by fit
|
|
||||||
|
|
||||||
### 1. Pre-flight tool guardrail (`tool_call` hook)
|
|
||||||
|
|
||||||
The documented hook example blocks `bash` commands containing `rm -rf` by
|
|
||||||
string match. Jev upgrades this to a semantic risk gate: state = the command +
|
|
||||||
cwd context; questions = Noul "destroys or exfiltrates data", Score "blast
|
|
||||||
radius", Choice over {allow, confirm, block}. Confidence-gate it: auto-allow
|
|
||||||
when safe > 0.9, ask the user in between, hard-block above a destruction
|
|
||||||
threshold. Same shape applies to `write`/`edit` paths (protected files) and to
|
|
||||||
`bash` chains that look innocent but end in a push.
|
|
||||||
|
|
||||||
Caveat: the model reads literally; keep the deterministic regex checks too —
|
|
||||||
Jev is the fuzzy layer on top, not the replacement.
|
|
||||||
|
|
||||||
### 2. Prompt-injection screen on tool results (`tool_result` hook)
|
|
||||||
|
|
||||||
Web pages, issue bodies, and arbitrary repo files can contain text engineered
|
|
||||||
to steer the agent. A `tool_result` handler can ask Jev: Noul "contains
|
|
||||||
instructions attempting to steer the agent or exfiltrate secrets", and if it
|
|
||||||
fires, annotate or redact the content before it enters context. Cost per
|
|
||||||
screening ~$0.0002 — cheap enough to run on every external read. Note the
|
|
||||||
docs' own warning cuts both ways: injection-shaped state can also move Jev's
|
|
||||||
answer, so treat the verdict as a tripwire for human-visible annotation, not
|
|
||||||
an autonomous veto.
|
|
||||||
|
|
||||||
### 3. Verification of agent output (the "verify everything" pattern)
|
|
||||||
|
|
||||||
An `agent_end` hook or a `verify` custom tool that scores the turn's outcome:
|
|
||||||
Noul per acceptance criterion ("the reported fix actually reproduces-and-tests
|
|
||||||
the changed path"), Score "evidence quality", Choice over
|
|
||||||
{complete, partial, off-track}. This is the System One workflow shape TypeSafe
|
|
||||||
built its evals around, and it plugs directly into the harness's verify-before-
|
|
||||||
yield discipline: an independent cheap judge over the transcript before the
|
|
||||||
turn is declared done.
|
|
||||||
|
|
||||||
### 4. Memory pipeline pre-filter (cheap Phase 1)
|
|
||||||
|
|
||||||
The local memory pipeline runs per-session extraction on the `default`-role
|
|
||||||
model for every changed session (cap: 64 sessions/startup, 4k-token inputs).
|
|
||||||
A Jev pre-pass over each session could ask: Noul "contains durable signal"
|
|
||||||
(decisions, constraints, resolved failures), Score "density of signal",
|
|
||||||
Noul "worth burning an LLM extraction on" — so the expensive extractor only
|
|
||||||
runs on sessions Jev rates. At 64 sessions x ~4k tokens the LLM pass is a few
|
|
||||||
cents; Jev's version is ~1% of that and ~100x faster, which is what makes
|
|
||||||
scanning *every* session every cycle affordable instead of capped.
|
|
||||||
|
|
||||||
### 5. Task routing / cascade (which agent, which model)
|
|
||||||
|
|
||||||
The `task` tool takes an `agent` choice (scout / task / reviewer /
|
|
||||||
security-reviewer) and `workpool` spreads items. A Jev front-end classifies
|
|
||||||
each item: Choice over agent types, Score "difficulty", Noul "read-only
|
|
||||||
research" — routing cheap items to scout, reserving expensive models. Same
|
|
||||||
cascade logic TypeSafe demos on support tickets, applied to subagent spawn.
|
|
||||||
This is advisory: a wrapper (custom tool or hook around `task`) makes the
|
|
||||||
call; the harness still owns control flow.
|
|
||||||
|
|
||||||
### 6. Config-drift guard on self-edits
|
|
||||||
|
|
||||||
When an agent writes to `AGENTS.md`, `RULES.md`, or a skill (the dreaming
|
|
||||||
pipeline in doc 03 does exactly this), a hook can gate the write: Noul "this
|
|
||||||
change encodes a one-off incident as a permanent rule", Noul "this rule
|
|
||||||
contradicts an existing rule in state". Cheap insurance against an agent
|
|
||||||
slowly cluttering its own configuration — the failure mode auto-learn systems
|
|
||||||
drift into.
|
|
||||||
|
|
||||||
### 7. Batch triage over history: the RSI dreaming loop
|
|
||||||
|
|
||||||
The biggest one gets its own document: replaying past sessions to mine
|
|
||||||
improvement opportunities (new skills, AGENTS.md/RULES.md edits, system-prompt
|
|
||||||
updates, new tools). Jev is what makes scanning *all* past interactions
|
|
||||||
economical. See [03-dreaming-with-jev.md](./03-dreaming-with-jev.md).
|
|
||||||
|
|
||||||
## Not Jev-shaped (do not force it)
|
|
||||||
|
|
||||||
- **Session titles, commit messages, summaries, compaction** — these are
|
|
||||||
*string generation*; the harness routes them to `tiny`/`smol` LLM roles for
|
|
||||||
good reason. Jev has no decoder; it cannot write them.
|
|
||||||
- **Code generation, refactors, reviews that need new text** — bounded-output
|
|
||||||
scoring can *rank* candidates but cannot produce them.
|
|
||||||
- **Anything needing knowledge outside the state you send** — Jev has no
|
|
||||||
world knowledge beyond state; no lookups. Retrieve first, judge second.
|
|
||||||
- **Image/screens understanding** — text-only state (as of Jev 1.13).
|
|
||||||
|
|
||||||
## Honest limits before betting on any of this
|
|
||||||
|
|
||||||
- Early access product; `jev-latest` currently resolves to `jev-1.13.0` and
|
|
||||||
moving GPU capacity means rate limits (250k tok/s, 1200 req/min) and
|
|
||||||
behavior can shift. Pin the version and log the response's `model` field.
|
|
||||||
- All performance numbers are TypeSafe's own, launch-week, unreproduced.
|
|
||||||
- Calibration ≠ correctness: validate every threshold against local outcomes
|
|
||||||
before letting it act autonomously.
|
|
||||||
- State ceiling: 64k tokens per call. Whole sessions don't fit; segment first
|
|
||||||
(doc 03 does).
|
|
||||||
@ -1,246 +0,0 @@
|
|||||||
# Dreaming with Jev: a self-improvement method for the pi configuration
|
|
||||||
|
|
||||||
*RSI project — proposal. Status: design, ready for staged implementation.*
|
|
||||||
|
|
||||||
## What "dreaming" is
|
|
||||||
|
|
||||||
Dreaming is a scheduled batch job that replays the agent's past interactions
|
|
||||||
and mines them for improvement opportunities, then turns the best ones into
|
|
||||||
configuration changes:
|
|
||||||
|
|
||||||
- new or updated **skills** (`~/.omp/agent/skills/<name>/SKILL.md`)
|
|
||||||
- **context file** edits (`~/.omp/agent/AGENTS.md`, `RULES.md`)
|
|
||||||
- **system prompt** updates (`APPEND_SYSTEM.md`, `PERSONALITY.md`)
|
|
||||||
- new **custom tools** (`~/.omp/agent/tools/*.ts`)
|
|
||||||
|
|
||||||
Nothing lands directly. Every output is a reviewable artifact — a Gitea PR
|
|
||||||
against the config repo and/or a pm-board proposal — and the user merges.
|
|
||||||
|
|
||||||
## Why Jev, and why now
|
|
||||||
|
|
||||||
The harness already has the raw material: every persisted session is a JSONL
|
|
||||||
file under `~/.omp/agent/sessions/<encoded-cwd>/`, plus prompt history in
|
|
||||||
`~/.omp/agent/history.db` (FTS5). The local memory pipeline (`memory.backend:
|
|
||||||
local`) already replays changed sessions — but it runs a `default`-role LLM per
|
|
||||||
session, caps at 64 sessions per startup, 300 scanned, with 12-hour idle
|
|
||||||
gating. It is an *understanding* pipeline, not an *improvement* pipeline.
|
|
||||||
|
|
||||||
The bottleneck is triage: deciding which parts of which sessions are worth a
|
|
||||||
$0.01+ LLM pass. That is a bounded judgment — "is there a durable improvement
|
|
||||||
opportunity in this segment, and what kind?" — exactly Jev's shape:
|
|
||||||
|
|
||||||
- input-only billing at $0.042/MTok, free output
|
|
||||||
- 70-500 ms per call, questions in one call evaluated in parallel
|
|
||||||
- calibrated probabilities → thresholds you can tune against outcomes
|
|
||||||
- every candidate question is a Noul/Score/Choice, no string parsing
|
|
||||||
|
|
||||||
At those economics the dreamer can screen **everything** (all sessions, all
|
|
||||||
segments) every night and escalate ~1-2% to an LLM. That inverts the usual
|
|
||||||
sampling-with-heuristics design: no session is skipped because it was cheap to
|
|
||||||
look.
|
|
||||||
|
|
||||||
## Pipeline
|
|
||||||
|
|
||||||
```
|
|
||||||
S0 harvest → S1 segment → S2 jev triage → S3 cluster → S4 LLM synthesis
|
|
||||||
↓
|
|
||||||
S6 calibration ← S5 human gate (PR + pm-board proposals)
|
|
||||||
```
|
|
||||||
|
|
||||||
### S0 — Harvest
|
|
||||||
|
|
||||||
Sources:
|
|
||||||
|
|
||||||
- `~/.omp/agent/sessions/<bucket>/*.jsonl` — parse per the session format:
|
|
||||||
fixed-width 256-byte title slot first, then session header, then entries.
|
|
||||||
Keep `message` entries (roles are camelCase: `user`, `assistant`,
|
|
||||||
`toolResult`); note `toolCall` blocks live inside assistant content.
|
|
||||||
Skip or downweight: `compaction`, `branch_summary`, `custom` (opaque),
|
|
||||||
entries after `reset_boundary`.
|
|
||||||
- `~/.omp/agent/history.db` (`history` table) — bare user prompts are a cheap
|
|
||||||
signal of repeated requests.
|
|
||||||
- pm-board `.pm/` state — accepted/rejected proposals tell you which past
|
|
||||||
suggestions the user actually wanted.
|
|
||||||
- Generated skills and `learned.md` under the memory root — what has already
|
|
||||||
been learned, so the dreamer proposes deltas, not duplicates.
|
|
||||||
|
|
||||||
### S1 — Segment
|
|
||||||
|
|
||||||
Cut each session into units small enough for one Jev call (state ceiling 64k
|
|
||||||
tokens; aim ≤4k). Natural cuts: one user intent → the assistant trajectory
|
|
||||||
that resolved it (tool calls + results + verdicts). Attach metadata: project
|
|
||||||
cwd, date, exit reason (`session_exit` entries), todo outcomes.
|
|
||||||
|
|
||||||
### S2 — Jev triage (the speculative fan-out)
|
|
||||||
|
|
||||||
One call per segment; all questions ride the same call:
|
|
||||||
|
|
||||||
```python
|
|
||||||
response = jev.system_one(
|
|
||||||
state={
|
|
||||||
"segment": seg.text, # redacted (see Safety)
|
|
||||||
"meta": {"cwd": seg.cwd, "date": seg.date, "exit": seg.exit_reason},
|
|
||||||
},
|
|
||||||
questions={
|
|
||||||
"friction": Noul("The user had to correct, repeat, or unblock the agent"),
|
|
||||||
"repeated": Noul("This is a routine workflow, done by hand, that recurs"),
|
|
||||||
"failure": Noul("The attempt failed or needed several retries"),
|
|
||||||
"config_gap": Noul("The failure or friction traces to a missing skill, rule, prompt guidance, or tool"),
|
|
||||||
"waste": Noul("The path taken was clearly inefficient vs a simpler one"),
|
|
||||||
"already_configured": Noul("The behavior that would fix this is already written in the provided current-config summary"),
|
|
||||||
"value": Score("How much lasting improvement is available here",
|
|
||||||
criteria=["Nothing actionable",
|
|
||||||
"Cosmetic preference",
|
|
||||||
"Workflow would improve one class of tasks",
|
|
||||||
"Recurring failure class would be prevented",
|
|
||||||
"Systemic issue affecting many future sessions"]),
|
|
||||||
"opportunity": Choice("What kind of change would fix it",
|
|
||||||
criteria={"new_skill": "A repeatable workflow deserves its own SKILL.md",
|
|
||||||
"rule_update": "A hard rule (RULES.md) or context guidance (AGENTS.md) is missing or wrong",
|
|
||||||
"prompt_update": "System-prompt behavior guidance should change (APPEND_SYSTEM.md)",
|
|
||||||
"new_tool": "A deterministic helper tool is missing",
|
|
||||||
"memory_only": "Worth remembering, not config-worthy",
|
|
||||||
"nothing": "No improvement available"}),
|
|
||||||
},
|
|
||||||
)
|
|
||||||
```
|
|
||||||
|
|
||||||
Pass a compact summary of the *current* config (skill names + descriptions,
|
|
||||||
rules titles, APPEND_SYSTEM.md) in state so `already_configured` can veto
|
|
||||||
duplicate proposals — this is the "context rot" safeguard: send only the fields
|
|
||||||
each question needs.
|
|
||||||
|
|
||||||
**Gating:** proceed only where `config_right_side` confidence is real:
|
|
||||||
|
|
||||||
```python
|
|
||||||
keep = (a["config_gap"].noul > 0.7 and a["already_configured"].noul < 0.3
|
|
||||||
and a["value"].score >= 2.0 and a["value"].confidence > 0.6)
|
|
||||||
```
|
|
||||||
|
|
||||||
Flat distributions ("value" confidence < 0.5) are a signal the criteria are
|
|
||||||
wrong for that segment — log them, do not guess. Thresholds live in one config
|
|
||||||
file; the calibration loop (S6) tunes them.
|
|
||||||
|
|
||||||
### S3 — Cluster and dedupe
|
|
||||||
|
|
||||||
Ten segments about git-push friction are one proposal, not ten. Group by
|
|
||||||
`opportunity_type`, then similarity over segment embeddings (or a second,
|
|
||||||
tiny Jev Choice over candidate groups — still bounded, still cheap). Keep the
|
|
||||||
top cluster exemplars (highest `value` score), cap proposals per dream run
|
|
||||||
(default 5) to bound blast radius.
|
|
||||||
|
|
||||||
### S4 — Synthesis (this is where the LLM earns its price)
|
|
||||||
|
|
||||||
Jev cannot write. Hand each surviving cluster — evidence links (session ids,
|
|
||||||
timestamps, the exact friction quotes) plus the relevant current-config files —
|
|
||||||
to a harness run (`omp -p --no-session`, the same pattern the pm-address timer
|
|
||||||
uses to reconcile boards). The LLM drafts one artifact per cluster:
|
|
||||||
|
|
||||||
| Cluster verdict | Artifact it drafts |
|
|
||||||
| --- | --- |
|
|
||||||
| `new_skill` | `<name>/SKILL.md` with `name` + `description` frontmatter (description required for native discovery), body as a workflow with evidence-linked examples |
|
|
||||||
| `rule_update` | diff for `~/.omp/agent/RULES.md` (hard rules only — short) or `AGENTS.md` (durable background) |
|
|
||||||
| `prompt_update` | diff for `APPEND_SYSTEM.md` |
|
|
||||||
| `new_tool` | `.omp/tools/<name>.ts` custom-tool module sketch (factory + Zod schema), marked needs-implementation |
|
|
||||||
|
|
||||||
### S5 — Human gate
|
|
||||||
|
|
||||||
Artifacts are written to a dream workspace branch:
|
|
||||||
|
|
||||||
- one Gitea PR per dream run against the config repo (per the standing branch
|
|
||||||
workflow: never commit to main; the user merges), and
|
|
||||||
- pm-board proposals in `.pm/proposals/*.md` (frontmatter `op:`/`id:`/
|
|
||||||
`section:`/`title:`) for board-tracked items.
|
|
||||||
|
|
||||||
The user's accept/reject is the ground-truth label for S6.
|
|
||||||
|
|
||||||
### S6 — Calibration loop
|
|
||||||
|
|
||||||
Log every verdict with its probability and its eventual outcome:
|
|
||||||
|
|
||||||
```json
|
|
||||||
{"segment": "...", "config_gap_noul": 0.83, "value_score": 2.4,
|
|
||||||
"cluster": "git-push-friction", "outcome": "accepted", "artifact": "PR #12"}
|
|
||||||
```
|
|
||||||
|
|
||||||
Then measure, per question, whether the 0.8 bucket really lands ~80% accepted.
|
|
||||||
This is RLCD's philosophy applied to the dreamer itself: confidence must track
|
|
||||||
outcomes, and drift in that curve is the alarm that thresholds (or the criteria
|
|
||||||
wording — Jev reads literally) need re-tuning. Calibration data is also what
|
|
||||||
justifies loosening the human gate later, if ever.
|
|
||||||
|
|
||||||
## Wiring into this harness
|
|
||||||
|
|
||||||
- **Scheduler:** mirror the pm-address pattern (hourly `omp -p --no-session`
|
|
||||||
reconciliation in `deploy/README.md`) — a nightly `dream` timer/cron that
|
|
||||||
invokes the pipeline, then a synthesis `omp` run over the shortlist.
|
|
||||||
- **Secrets:** the memory pipeline redacts secret patterns before writing
|
|
||||||
memory artifacts; the dreamer must do the same *before* anything leaves the
|
|
||||||
machine as Jev state, and re-redact before synthesis.
|
|
||||||
- **Overlap with memory:** memory consolidation produces knowledge summaries;
|
|
||||||
dreaming produces config changes. They share S0. If `memory.backend: local`
|
|
||||||
is enabled, reuse its per-session extraction cadence markers instead of
|
|
||||||
re-parsing unchanged sessions (the pipeline tracks processed-since mtime).
|
|
||||||
- **Where outputs land:** skills into `~/.omp/agent/skills/<name>/` (native
|
|
||||||
provider, priority 100; beats same-named lower providers), rules per the
|
|
||||||
RULES.md contract (short, sticky, native locations only), tools into
|
|
||||||
`~/.omp/agent/tools/`.
|
|
||||||
|
|
||||||
## Cost model
|
|
||||||
|
|
||||||
Per dream run (rough, at self-reported prices):
|
|
||||||
|
|
||||||
| Stage | Volume | Cost |
|
|
||||||
| --- | --- | --- |
|
|
||||||
| S2 Jev: 300 sessions → ~1,200 segments x ~4k tokens | ~5 MTok input | **~$0.21** |
|
|
||||||
| S4 LLM synthesis: ~5 clusters x ~15k tokens | ~75k tok | cents–$1 |
|
|
||||||
| S0-S3, S6 code + SQLite | — | negligible |
|
|
||||||
|
|
||||||
A full-history dream costs well under $1. The LLM synthesis spend is the
|
|
||||||
larger line item and scales with proposal count, which S3 caps.
|
|
||||||
|
|
||||||
## Safety: this is self-modification, treat it like it
|
|
||||||
|
|
||||||
1. **Human gate.** No direct writes to live config. PRs + proposals only; the
|
|
||||||
user merges. (Also required by the house rule: no commits to main.)
|
|
||||||
2. **Blast-radius cap.** ≤5 proposals per run, each with evidence links; a
|
|
||||||
proposal without a session citation is rejected by construction.
|
|
||||||
3. **Injection.** Sessions contain untrusted content (web pages, issue text)
|
|
||||||
and "state is not hostile" applies — content can argue for its own
|
|
||||||
improvement. Mitigations: redaction before Jev sees it, the LLM synthesizer
|
|
||||||
independently reviews evidence, and the human gate sees the evidence too.
|
|
||||||
4. **Feedback loops.** The dreamer must not grade its own artifacts as
|
|
||||||
"accepted" — outcomes come from the human's merge/reject actions only.
|
|
||||||
5. **Reversibility.** Config lives in git; every accepted proposal is a
|
|
||||||
reviewable commit; `RULES.md` stays short so the sticky layer can't bloat.
|
|
||||||
|
|
||||||
## Implementation notes
|
|
||||||
|
|
||||||
- Adapter interface first: `JevClient` with three backends — `jev` (real API,
|
|
||||||
needs `TYPESAFE_API_KEY` early access, or Vercel AI gateway), `llm` (a
|
|
||||||
frontier LLM constrained to the same schema, e.g. via TypeSafe's published
|
|
||||||
`system-one-adapter-python` shim), and `mock` (deterministic, for tests).
|
|
||||||
The interface is the design; the backend is swappable, so the pipeline can
|
|
||||||
be built and audited before the key exists.
|
|
||||||
- Pin `jev-1.13.0` in config, log the `model` field from every response.
|
|
||||||
- Dry-run mode writes an audit report (what would have been proposed, with
|
|
||||||
Jev's raw probabilities) and nothing else.
|
|
||||||
|
|
||||||
## Rollout
|
|
||||||
|
|
||||||
| Milestone | Gate |
|
|
||||||
| --- | --- |
|
|
||||||
| M0: parser + S2 dry run over last 30 days, human reads the audit report | report is sensibly targeted (eyeball) |
|
|
||||||
| M1: full pipeline, thresholds strict (`config_gap > 0.8`), ≤3 proposals/run | user accepts ≥1 real artifact |
|
|
||||||
| M2: weekly cadence, thresholds tuned from calibration data | calibration curve within ±15pp of diagonal |
|
|
||||||
|
|
||||||
## Open questions
|
|
||||||
|
|
||||||
- Does Jev's calibration hold on *this* corpus (session transcripts are
|
|
||||||
long, messy, and self-authored)? M0 exists to answer this before anything
|
|
||||||
acts.
|
|
||||||
- Cluster-level evidence may need a second Jev pass with the current config in
|
|
||||||
state (serial call, justified dependency per the primitives' independence
|
|
||||||
rule).
|
|
||||||
- Whether S4 synthesis should also run the doc-02 "config-drift guard" on its
|
|
||||||
own drafts before proposing them.
|
|
||||||
Loading…
x
Reference in New Issue
Block a user