Compare commits
1 Commits
main
...
chore/pm-r
| Author | SHA1 | Date | |
|---|---|---|---|
|
5202bd98be |
7
.gitignore
vendored
7
.gitignore
vendored
@ -1,2 +1,9 @@
|
||||
__pycache__/
|
||||
*.pyc
|
||||
|
||||
# pm-board runner output and legacy runtime files (never tracked)
|
||||
.pm/runs.jsonl
|
||||
.pm/address-state.json
|
||||
.pm/address-history.jsonl
|
||||
.pm/feedback-state.json
|
||||
.pm/outbox.md
|
||||
|
||||
@ -1,3 +1,4 @@
|
||||
gitea_url: "https://git.jajahome.duckdns.org/omp/rsi"
|
||||
requirements:
|
||||
- id: R1
|
||||
title: Jev research summary
|
||||
|
||||
@ -1 +0,0 @@
|
||||
https://git.jajahome.duckdns.org/omp/rsi
|
||||
@ -1,211 +0,0 @@
|
||||
# Jev: what it does and how it works
|
||||
|
||||
Research summary of TypeSafe AI's Jev, launched 15 September 2026. Written so an
|
||||
undergraduate with one AI course behind them can follow it. Sources at the
|
||||
bottom; claims are tagged confirmed, self-reported, or speculative.
|
||||
|
||||
## TL;DR
|
||||
|
||||
Jev is not a chatbot and not a smaller LLM. It is a **decision model**: you send
|
||||
it a piece of state (text or JSON) plus a set of pre-defined questions, and it
|
||||
returns typed answers — a category, a number on a scale, or a probability — each
|
||||
with calibrated probabilities. It never generates text. That single design
|
||||
choice makes it ~100x faster and ~100-400x cheaper than a frontier LLM on these
|
||||
narrow questions, and makes malformed answers structurally impossible.
|
||||
|
||||
## The problem it solves
|
||||
|
||||
A normal LLM returns **strings**. To use a string inside software you must
|
||||
parse it, validate it, and hope the model did not go off the rails. Three costs
|
||||
follow:
|
||||
|
||||
1. **Latency.** LLMs generate one token at a time, each conditioned on the
|
||||
last. A 500-token answer is ~500 sequential forward passes. Frontier
|
||||
responses take 3s to 329s end to end.
|
||||
2. **Cost.** Output tokens cost ~5x input tokens because each one is another
|
||||
decode pass.
|
||||
3. **Type errors and hallucination.** No matter how smart the model, the output
|
||||
space is "any string", so it can emit an unexpected value, malformed JSON,
|
||||
or a made-up fact. When the consumer is software — not a human reading it —
|
||||
that is a deal-breaker, especially deep in a dependency chain.
|
||||
|
||||
TypeSafe's thesis: most *machine-consumed* intelligence is not "write me an
|
||||
essay". It is small judgments — which bucket, how severe, is this true — that
|
||||
software could consume directly if the model were shaped for it. Kahneman's
|
||||
System 1 / System 2 distinction is the branding: Jev is fast intuition, LLMs
|
||||
are slow deliberation.
|
||||
|
||||
## The interface
|
||||
|
||||
```
|
||||
state + typed questions → typed probabilistic answers
|
||||
```
|
||||
|
||||
One endpoint (`POST https://api.typesafe.ai/v1/systemone`), SDKs in Python
|
||||
(`typesafe-sdk`) and TypeScript (`@typesafe-ai/sdk`). `state` is a string, JSON
|
||||
object, or array. Limits: 64k tokens for state + all questions, text only (no
|
||||
images).
|
||||
|
||||
### Example
|
||||
|
||||
```python
|
||||
from typesafe_sdk import Choice, Noul, Score, TypeSafeClient
|
||||
|
||||
client = TypeSafeClient()
|
||||
|
||||
response = client.system_one(
|
||||
state={
|
||||
"ticket": {"subject": "Duplicate charge",
|
||||
"messages": [{"from": "customer",
|
||||
"text": "I was charged twice for order A-104."}]},
|
||||
"refund_policy": "Duplicate charges are eligible for a refund.",
|
||||
},
|
||||
questions={
|
||||
"department": Choice(instructions="Which team should handle this",
|
||||
criteria={"billing": "Payment or subscription issues",
|
||||
"technical": "Bugs or integration problems",
|
||||
"sales": "Pricing or account questions"}),
|
||||
"frustration": Score(instructions="How frustrated the customer appears",
|
||||
criteria=["Calm, just stating facts",
|
||||
"Frustrated but civil",
|
||||
"Very angry, strong language"]),
|
||||
"refund_wanted": Noul(instructions="The customer explicitly asks for a refund"),
|
||||
"policy_supports": Noul(instructions="The stated refund policy covers this"),
|
||||
},
|
||||
)
|
||||
print(response.answers["department"].choice, response.answers["department"].confidence)
|
||||
```
|
||||
|
||||
One call returns every answer in 70-500 ms.
|
||||
|
||||
## The three primitives
|
||||
|
||||
| Primitive | Question it answers | Returns |
|
||||
| --- | --- | --- |
|
||||
| **Noul** | "Is this true?" | a 0-1 probability. No separate confidence field — the number *is* the belief. |
|
||||
| **Choice** | "Which of these?" | winning option + probability for every option (up to 255) + confidence |
|
||||
| **Score** | "Where on this scale?" | a value that can land *between* levels (e.g. 1.35) + per-level probabilities + confidence |
|
||||
|
||||
Design rules from the docs, worth internalizing:
|
||||
|
||||
- Define Choice criteria as concrete situations, and always include an `other`
|
||||
option so the model can say "nothing fits" instead of picking the closest
|
||||
wrong thing.
|
||||
- Score levels should be concrete situations ("service down, no workaround"),
|
||||
not vague adjectives ("high").
|
||||
- Questions in one call are evaluated **independently** against the same state.
|
||||
A tenth question costs a few tokens and almost no latency, so the winning
|
||||
pattern is *speculative fan-out*: ask everything, let code ignore what does
|
||||
not apply. If question B truly depends on question A's answer, make a second
|
||||
call with updated state.
|
||||
|
||||
## How it works
|
||||
|
||||
### Why one forward pass is possible
|
||||
|
||||
An LLM must decode sequentially because its answer space is effectively
|
||||
unbounded (any string). Jev's answer space is **fixed and enumerated in
|
||||
advance** — 255 options, a scale, or a boolean. A bounded answer space can be
|
||||
*scored* rather than generated: one forward pass computes a probability for
|
||||
every possible answer, no decode loop afterward.
|
||||
|
||||
That one fact explains three marketing claims at once:
|
||||
|
||||
- **Speed** (70-500 ms): one pass instead of hundreds. This is different from
|
||||
Transformer training parallelism — LLM *inference* was always sequential,
|
||||
and Jev parallelizes inference-time decision computation for bounded
|
||||
outputs only.
|
||||
- **Free output tokens** ($0.042/MTok input, $0 output): there is nothing
|
||||
after the single pass to meter.
|
||||
- **"Cannot hallucinate"**: outputs are members of a predefined set by
|
||||
construction; a malformed string was never a possible output. Careful: this
|
||||
means *no schema errors*, not *always correct*. Jev can confidently pick the
|
||||
wrong category. "Calibration" and "correctness" are different properties.
|
||||
|
||||
What the architecture actually is — encoder-only transformer with
|
||||
classification heads, diffusion variant, something purpose-built — is
|
||||
**undisclosed**. Treat all such explanations as informed speculation.
|
||||
|
||||
### RLCD: training for honesty about uncertainty
|
||||
|
||||
TypeSafe's training method, RLCD (Reinforcement Learning for Calibrated
|
||||
Decisions), sits beside its two better-known cousins:
|
||||
|
||||
| Method | Reward signal | Used for |
|
||||
| --- | --- | --- |
|
||||
| RLHF | human raters prefer this output | chat models (the lineage Jev's founder, Diogo Almeida, worked on at OpenAI) |
|
||||
| RLVR | output passes a programmatic check | math/code reasoning |
|
||||
| **RLCD** | stated confidence matches actual accuracy | Jev |
|
||||
|
||||
Calibration, concretely: a weather forecaster who says "70% chance of rain" on
|
||||
100 days should get rain on roughly 70 of them. RLCD rewards the *match*
|
||||
between confidence and outcomes, not persuasiveness. The payoff for software:
|
||||
you can branch on the number. Act automatically at 0.95, ask a human at 0.55 —
|
||||
but only if the number is honest *in aggregate*.
|
||||
|
||||
Two honest caveats. First, calibration is a distribution property: an
|
||||
individual 0.9 answer can still be wrong. Second, TypeSafe's accuracy claims
|
||||
come from their own "workflow evals" (agreement with the average of two
|
||||
frontier LLMs), not from a public reproducible benchmark. The latency/cost
|
||||
mechanism stands on its own logic; the accuracy numbers are so far self-run.
|
||||
|
||||
## Worked examples
|
||||
|
||||
**1. Support-ticket triage (the canonical case).** The code example above:
|
||||
category, frustration, refund intent, and policy coverage answered in one
|
||||
call; Python branches on the answers. On a million tickets, the launch-week
|
||||
arithmetic gives ~$6.5k instead of ~$30k with most tickets answered in under
|
||||
half a second.
|
||||
|
||||
**2. Code review risk matrix** (`devagrawal09/jev-review`). A staged reviewer:
|
||||
first a Noul risk matrix over a diff, then Choice/Score profiles per file, then
|
||||
severity. Cheap filters decide which files a big model looks at.
|
||||
|
||||
**3. Browser agent** (`browser-use/jev-ultrafast`). Each page becomes a
|
||||
numbered element table; one Jev call picks both the operation (CLICK / TYPE /
|
||||
SCROLL / DONE / BLOCKED) and its target; a small LLM runs only to type text.
|
||||
Booked a flight on real Google Flights in 7.1 s for $0.0039.
|
||||
|
||||
**4. Playing Doom.** Game state as a structured object in, action Choice out,
|
||||
~10 decisions/second. Games are the clean demo of *judgment per tick* — the
|
||||
same shape as a trading bot deciding per 300 ms block or a drone's ~2.5 Hz
|
||||
tactical layer sitting above a code-owned flight controller.
|
||||
|
||||
**5. Economics at scale** (1kpapers.com). 1,018 papers summarized by a
|
||||
generative model ($3.99), classified into 24 topics by Jev — **$0.08**, median
|
||||
256 ms per paper. Generative model writes, decision model classifies.
|
||||
|
||||
## What Jev is bad at (per TypeSafe's own "jaggedness" page)
|
||||
|
||||
- **Reads literally.** It answers the question you wrote, not the one you
|
||||
meant. Negations and implied conditions land at face value.
|
||||
- **Not a calculator.** Counting, date arithmetic, anything orderable-as-text
|
||||
is unreliable. Iterate in code, ask one Noul per item.
|
||||
- **Context rot.** Accuracy drops when state is padded with material the
|
||||
question does not need. Retrieve and filter first.
|
||||
- **State is not hostile.** Text engineered to argue for its own classification
|
||||
can move the answer. User-controlled content in state is a threat model you
|
||||
must handle.
|
||||
- **Does not generate.** No strings, no code, no summaries. If you need a
|
||||
value extracted from free text, get candidates with code or a generative
|
||||
model, then let Jev pick.
|
||||
- It also reads *literally*: contradictory criteria confuse it.
|
||||
|
||||
## Status of the claims
|
||||
|
||||
| Claim | Status |
|
||||
| --- | --- |
|
||||
| Three primitives, one parallel pass, RLCD by name | confirmed (TypeSafe's own materials) |
|
||||
| RLCD loss/reward construction, architecture | undisclosed |
|
||||
| $0.042/MTok, 70-500 ms, 20-200x / 40-400x gains | self-reported, not independently verified |
|
||||
| Workflow-eval accuracy vs GPT-6 Astra / Fable 5.1 average | internal methodology, no public ground truth; one chart showed Jev below Sonnet 5 raw accuracy |
|
||||
| "Zero hallucinations" | = zero schema/type errors by construction; semantic errors still possible |
|
||||
|
||||
## Sources
|
||||
|
||||
- TypeSafe launch post: <https://typesafe.ai/blog/introducing-system-one-models-and-jev>
|
||||
- Docs: <https://docs.typesafe.ai/>
|
||||
- Mechanism explainer (RLCD, single-pass, confirmed-vs-speculative): <https://www.explainx.ai/blog/how-does-jev-work-rlcd-system-one-model-explained-2026>
|
||||
- Practical guide with shipped demos and failure modes: <https://dev.to/valyuai/how-to-use-jev-a-practical-guide-to-typesafes-system-one-model-g5e>
|
||||
- The Register's critical writeup: <https://www.theregister.com/ai-and-ml/2026/09/16/typesafe-ai-debuts-model-for-machines-that-plays-doom/5296711>
|
||||
- Project reference gist: <https://gist.github.com/pjburnhill/adf8d28efcad9df037bfdece178ef965>
|
||||
@ -1,120 +0,0 @@
|
||||
# Jev in the pi (Oh My Pi) harness
|
||||
|
||||
Use cases for a decision model inside a coding-agent harness. Every claim below
|
||||
is grounded in the harness's own docs (`omp://`); Jev facts come from
|
||||
[01-what-is-jev.md](./01-what-is-jev.md). The mental filter for each idea: Jev
|
||||
answers *bounded* questions ("is this X?", "which bucket?", "how severe?") in
|
||||
~100-500 ms for ~$0.0002. The harness's LLM does the thinking and the writing.
|
||||
|
||||
## Where the harness can host a decision model
|
||||
|
||||
| Surface | Mechanism | File location |
|
||||
| --- | --- | --- |
|
||||
| Hooks / extensions | event bus with `tool_call`, `tool_result`, `context`, `before_agent_start`, `session_*` handlers | `<cwd>/.omp/hooks/pre/*.ts`, `~/.omp/agent/hooks/pre/*.ts` |
|
||||
| Custom tools | model-callable functions, Zod/arktype schemas | `~/.omp/agent/tools`, `.omp/tools` |
|
||||
| Skills | `SKILL.md` capability packs | `~/.omp/agent/skills/<name>/` (also `managed-skills` via `autolearn.enabled`) |
|
||||
| Context files | `AGENTS.md` (user + project), `RULES.md` (sticky) | `~/.omp/agent/AGENTS.md`, `.omp/AGENTS.md` |
|
||||
| System prompt | `SYSTEM.md` (replace), `APPEND_SYSTEM.md` (add), `PERSONALITY.md` | `~/.omp/agent/...` |
|
||||
| Memory pipeline | two-phase extraction+consolidation over past sessions | `memory.backend: local`, phases run on `default` / `smol` model roles |
|
||||
| Raw material | persisted sessions (JSONL), prompt history (SQLite FTS5) | `~/.omp/agent/sessions/.../*.jsonl`, `~/.omp/agent/history.db` |
|
||||
| Model routing | `models.yml`, roles (`smol`, `tiny`, `task`, `advisor`, ...) | `~/.omp/agent/models.yml` |
|
||||
|
||||
Jev is **not** a model provider in `models.yml` — it cannot chat, stream, or
|
||||
hold a conversation, so it cannot be a session model. It plugs in through
|
||||
hooks, custom tools, and batch jobs (see
|
||||
[03-dreaming-with-jev.md](./03-dreaming-with-jev.md)).
|
||||
|
||||
## Use cases, ranked by fit
|
||||
|
||||
### 1. Pre-flight tool guardrail (`tool_call` hook)
|
||||
|
||||
The documented hook example blocks `bash` commands containing `rm -rf` by
|
||||
string match. Jev upgrades this to a semantic risk gate: state = the command +
|
||||
cwd context; questions = Noul "destroys or exfiltrates data", Score "blast
|
||||
radius", Choice over {allow, confirm, block}. Confidence-gate it: auto-allow
|
||||
when safe > 0.9, ask the user in between, hard-block above a destruction
|
||||
threshold. Same shape applies to `write`/`edit` paths (protected files) and to
|
||||
`bash` chains that look innocent but end in a push.
|
||||
|
||||
Caveat: the model reads literally; keep the deterministic regex checks too —
|
||||
Jev is the fuzzy layer on top, not the replacement.
|
||||
|
||||
### 2. Prompt-injection screen on tool results (`tool_result` hook)
|
||||
|
||||
Web pages, issue bodies, and arbitrary repo files can contain text engineered
|
||||
to steer the agent. A `tool_result` handler can ask Jev: Noul "contains
|
||||
instructions attempting to steer the agent or exfiltrate secrets", and if it
|
||||
fires, annotate or redact the content before it enters context. Cost per
|
||||
screening ~$0.0002 — cheap enough to run on every external read. Note the
|
||||
docs' own warning cuts both ways: injection-shaped state can also move Jev's
|
||||
answer, so treat the verdict as a tripwire for human-visible annotation, not
|
||||
an autonomous veto.
|
||||
|
||||
### 3. Verification of agent output (the "verify everything" pattern)
|
||||
|
||||
An `agent_end` hook or a `verify` custom tool that scores the turn's outcome:
|
||||
Noul per acceptance criterion ("the reported fix actually reproduces-and-tests
|
||||
the changed path"), Score "evidence quality", Choice over
|
||||
{complete, partial, off-track}. This is the System One workflow shape TypeSafe
|
||||
built its evals around, and it plugs directly into the harness's verify-before-
|
||||
yield discipline: an independent cheap judge over the transcript before the
|
||||
turn is declared done.
|
||||
|
||||
### 4. Memory pipeline pre-filter (cheap Phase 1)
|
||||
|
||||
The local memory pipeline runs per-session extraction on the `default`-role
|
||||
model for every changed session (cap: 64 sessions/startup, 4k-token inputs).
|
||||
A Jev pre-pass over each session could ask: Noul "contains durable signal"
|
||||
(decisions, constraints, resolved failures), Score "density of signal",
|
||||
Noul "worth burning an LLM extraction on" — so the expensive extractor only
|
||||
runs on sessions Jev rates. At 64 sessions x ~4k tokens the LLM pass is a few
|
||||
cents; Jev's version is ~1% of that and ~100x faster, which is what makes
|
||||
scanning *every* session every cycle affordable instead of capped.
|
||||
|
||||
### 5. Task routing / cascade (which agent, which model)
|
||||
|
||||
The `task` tool takes an `agent` choice (scout / task / reviewer /
|
||||
security-reviewer) and `workpool` spreads items. A Jev front-end classifies
|
||||
each item: Choice over agent types, Score "difficulty", Noul "read-only
|
||||
research" — routing cheap items to scout, reserving expensive models. Same
|
||||
cascade logic TypeSafe demos on support tickets, applied to subagent spawn.
|
||||
This is advisory: a wrapper (custom tool or hook around `task`) makes the
|
||||
call; the harness still owns control flow.
|
||||
|
||||
### 6. Config-drift guard on self-edits
|
||||
|
||||
When an agent writes to `AGENTS.md`, `RULES.md`, or a skill (the dreaming
|
||||
pipeline in doc 03 does exactly this), a hook can gate the write: Noul "this
|
||||
change encodes a one-off incident as a permanent rule", Noul "this rule
|
||||
contradicts an existing rule in state". Cheap insurance against an agent
|
||||
slowly cluttering its own configuration — the failure mode auto-learn systems
|
||||
drift into.
|
||||
|
||||
### 7. Batch triage over history: the RSI dreaming loop
|
||||
|
||||
The biggest one gets its own document: replaying past sessions to mine
|
||||
improvement opportunities (new skills, AGENTS.md/RULES.md edits, system-prompt
|
||||
updates, new tools). Jev is what makes scanning *all* past interactions
|
||||
economical. See [03-dreaming-with-jev.md](./03-dreaming-with-jev.md).
|
||||
|
||||
## Not Jev-shaped (do not force it)
|
||||
|
||||
- **Session titles, commit messages, summaries, compaction** — these are
|
||||
*string generation*; the harness routes them to `tiny`/`smol` LLM roles for
|
||||
good reason. Jev has no decoder; it cannot write them.
|
||||
- **Code generation, refactors, reviews that need new text** — bounded-output
|
||||
scoring can *rank* candidates but cannot produce them.
|
||||
- **Anything needing knowledge outside the state you send** — Jev has no
|
||||
world knowledge beyond state; no lookups. Retrieve first, judge second.
|
||||
- **Image/screens understanding** — text-only state (as of Jev 1.13).
|
||||
|
||||
## Honest limits before betting on any of this
|
||||
|
||||
- Early access product; `jev-latest` currently resolves to `jev-1.13.0` and
|
||||
moving GPU capacity means rate limits (250k tok/s, 1200 req/min) and
|
||||
behavior can shift. Pin the version and log the response's `model` field.
|
||||
- All performance numbers are TypeSafe's own, launch-week, unreproduced.
|
||||
- Calibration ≠ correctness: validate every threshold against local outcomes
|
||||
before letting it act autonomously.
|
||||
- State ceiling: 64k tokens per call. Whole sessions don't fit; segment first
|
||||
(doc 03 does).
|
||||
@ -1,246 +0,0 @@
|
||||
# Dreaming with Jev: a self-improvement method for the pi configuration
|
||||
|
||||
*RSI project — proposal. Status: design, ready for staged implementation.*
|
||||
|
||||
## What "dreaming" is
|
||||
|
||||
Dreaming is a scheduled batch job that replays the agent's past interactions
|
||||
and mines them for improvement opportunities, then turns the best ones into
|
||||
configuration changes:
|
||||
|
||||
- new or updated **skills** (`~/.omp/agent/skills/<name>/SKILL.md`)
|
||||
- **context file** edits (`~/.omp/agent/AGENTS.md`, `RULES.md`)
|
||||
- **system prompt** updates (`APPEND_SYSTEM.md`, `PERSONALITY.md`)
|
||||
- new **custom tools** (`~/.omp/agent/tools/*.ts`)
|
||||
|
||||
Nothing lands directly. Every output is a reviewable artifact — a Gitea PR
|
||||
against the config repo and/or a pm-board proposal — and the user merges.
|
||||
|
||||
## Why Jev, and why now
|
||||
|
||||
The harness already has the raw material: every persisted session is a JSONL
|
||||
file under `~/.omp/agent/sessions/<encoded-cwd>/`, plus prompt history in
|
||||
`~/.omp/agent/history.db` (FTS5). The local memory pipeline (`memory.backend:
|
||||
local`) already replays changed sessions — but it runs a `default`-role LLM per
|
||||
session, caps at 64 sessions per startup, 300 scanned, with 12-hour idle
|
||||
gating. It is an *understanding* pipeline, not an *improvement* pipeline.
|
||||
|
||||
The bottleneck is triage: deciding which parts of which sessions are worth a
|
||||
$0.01+ LLM pass. That is a bounded judgment — "is there a durable improvement
|
||||
opportunity in this segment, and what kind?" — exactly Jev's shape:
|
||||
|
||||
- input-only billing at $0.042/MTok, free output
|
||||
- 70-500 ms per call, questions in one call evaluated in parallel
|
||||
- calibrated probabilities → thresholds you can tune against outcomes
|
||||
- every candidate question is a Noul/Score/Choice, no string parsing
|
||||
|
||||
At those economics the dreamer can screen **everything** (all sessions, all
|
||||
segments) every night and escalate ~1-2% to an LLM. That inverts the usual
|
||||
sampling-with-heuristics design: no session is skipped because it was cheap to
|
||||
look.
|
||||
|
||||
## Pipeline
|
||||
|
||||
```
|
||||
S0 harvest → S1 segment → S2 jev triage → S3 cluster → S4 LLM synthesis
|
||||
↓
|
||||
S6 calibration ← S5 human gate (PR + pm-board proposals)
|
||||
```
|
||||
|
||||
### S0 — Harvest
|
||||
|
||||
Sources:
|
||||
|
||||
- `~/.omp/agent/sessions/<bucket>/*.jsonl` — parse per the session format:
|
||||
fixed-width 256-byte title slot first, then session header, then entries.
|
||||
Keep `message` entries (roles are camelCase: `user`, `assistant`,
|
||||
`toolResult`); note `toolCall` blocks live inside assistant content.
|
||||
Skip or downweight: `compaction`, `branch_summary`, `custom` (opaque),
|
||||
entries after `reset_boundary`.
|
||||
- `~/.omp/agent/history.db` (`history` table) — bare user prompts are a cheap
|
||||
signal of repeated requests.
|
||||
- pm-board `.pm/` state — accepted/rejected proposals tell you which past
|
||||
suggestions the user actually wanted.
|
||||
- Generated skills and `learned.md` under the memory root — what has already
|
||||
been learned, so the dreamer proposes deltas, not duplicates.
|
||||
|
||||
### S1 — Segment
|
||||
|
||||
Cut each session into units small enough for one Jev call (state ceiling 64k
|
||||
tokens; aim ≤4k). Natural cuts: one user intent → the assistant trajectory
|
||||
that resolved it (tool calls + results + verdicts). Attach metadata: project
|
||||
cwd, date, exit reason (`session_exit` entries), todo outcomes.
|
||||
|
||||
### S2 — Jev triage (the speculative fan-out)
|
||||
|
||||
One call per segment; all questions ride the same call:
|
||||
|
||||
```python
|
||||
response = jev.system_one(
|
||||
state={
|
||||
"segment": seg.text, # redacted (see Safety)
|
||||
"meta": {"cwd": seg.cwd, "date": seg.date, "exit": seg.exit_reason},
|
||||
},
|
||||
questions={
|
||||
"friction": Noul("The user had to correct, repeat, or unblock the agent"),
|
||||
"repeated": Noul("This is a routine workflow, done by hand, that recurs"),
|
||||
"failure": Noul("The attempt failed or needed several retries"),
|
||||
"config_gap": Noul("The failure or friction traces to a missing skill, rule, prompt guidance, or tool"),
|
||||
"waste": Noul("The path taken was clearly inefficient vs a simpler one"),
|
||||
"already_configured": Noul("The behavior that would fix this is already written in the provided current-config summary"),
|
||||
"value": Score("How much lasting improvement is available here",
|
||||
criteria=["Nothing actionable",
|
||||
"Cosmetic preference",
|
||||
"Workflow would improve one class of tasks",
|
||||
"Recurring failure class would be prevented",
|
||||
"Systemic issue affecting many future sessions"]),
|
||||
"opportunity": Choice("What kind of change would fix it",
|
||||
criteria={"new_skill": "A repeatable workflow deserves its own SKILL.md",
|
||||
"rule_update": "A hard rule (RULES.md) or context guidance (AGENTS.md) is missing or wrong",
|
||||
"prompt_update": "System-prompt behavior guidance should change (APPEND_SYSTEM.md)",
|
||||
"new_tool": "A deterministic helper tool is missing",
|
||||
"memory_only": "Worth remembering, not config-worthy",
|
||||
"nothing": "No improvement available"}),
|
||||
},
|
||||
)
|
||||
```
|
||||
|
||||
Pass a compact summary of the *current* config (skill names + descriptions,
|
||||
rules titles, APPEND_SYSTEM.md) in state so `already_configured` can veto
|
||||
duplicate proposals — this is the "context rot" safeguard: send only the fields
|
||||
each question needs.
|
||||
|
||||
**Gating:** proceed only where `config_right_side` confidence is real:
|
||||
|
||||
```python
|
||||
keep = (a["config_gap"].noul > 0.7 and a["already_configured"].noul < 0.3
|
||||
and a["value"].score >= 2.0 and a["value"].confidence > 0.6)
|
||||
```
|
||||
|
||||
Flat distributions ("value" confidence < 0.5) are a signal the criteria are
|
||||
wrong for that segment — log them, do not guess. Thresholds live in one config
|
||||
file; the calibration loop (S6) tunes them.
|
||||
|
||||
### S3 — Cluster and dedupe
|
||||
|
||||
Ten segments about git-push friction are one proposal, not ten. Group by
|
||||
`opportunity_type`, then similarity over segment embeddings (or a second,
|
||||
tiny Jev Choice over candidate groups — still bounded, still cheap). Keep the
|
||||
top cluster exemplars (highest `value` score), cap proposals per dream run
|
||||
(default 5) to bound blast radius.
|
||||
|
||||
### S4 — Synthesis (this is where the LLM earns its price)
|
||||
|
||||
Jev cannot write. Hand each surviving cluster — evidence links (session ids,
|
||||
timestamps, the exact friction quotes) plus the relevant current-config files —
|
||||
to a harness run (`omp -p --no-session`, the same pattern the pm-address timer
|
||||
uses to reconcile boards). The LLM drafts one artifact per cluster:
|
||||
|
||||
| Cluster verdict | Artifact it drafts |
|
||||
| --- | --- |
|
||||
| `new_skill` | `<name>/SKILL.md` with `name` + `description` frontmatter (description required for native discovery), body as a workflow with evidence-linked examples |
|
||||
| `rule_update` | diff for `~/.omp/agent/RULES.md` (hard rules only — short) or `AGENTS.md` (durable background) |
|
||||
| `prompt_update` | diff for `APPEND_SYSTEM.md` |
|
||||
| `new_tool` | `.omp/tools/<name>.ts` custom-tool module sketch (factory + Zod schema), marked needs-implementation |
|
||||
|
||||
### S5 — Human gate
|
||||
|
||||
Artifacts are written to a dream workspace branch:
|
||||
|
||||
- one Gitea PR per dream run against the config repo (per the standing branch
|
||||
workflow: never commit to main; the user merges), and
|
||||
- pm-board proposals in `.pm/proposals/*.md` (frontmatter `op:`/`id:`/
|
||||
`section:`/`title:`) for board-tracked items.
|
||||
|
||||
The user's accept/reject is the ground-truth label for S6.
|
||||
|
||||
### S6 — Calibration loop
|
||||
|
||||
Log every verdict with its probability and its eventual outcome:
|
||||
|
||||
```json
|
||||
{"segment": "...", "config_gap_noul": 0.83, "value_score": 2.4,
|
||||
"cluster": "git-push-friction", "outcome": "accepted", "artifact": "PR #12"}
|
||||
```
|
||||
|
||||
Then measure, per question, whether the 0.8 bucket really lands ~80% accepted.
|
||||
This is RLCD's philosophy applied to the dreamer itself: confidence must track
|
||||
outcomes, and drift in that curve is the alarm that thresholds (or the criteria
|
||||
wording — Jev reads literally) need re-tuning. Calibration data is also what
|
||||
justifies loosening the human gate later, if ever.
|
||||
|
||||
## Wiring into this harness
|
||||
|
||||
- **Scheduler:** mirror the pm-address pattern (hourly `omp -p --no-session`
|
||||
reconciliation in `deploy/README.md`) — a nightly `dream` timer/cron that
|
||||
invokes the pipeline, then a synthesis `omp` run over the shortlist.
|
||||
- **Secrets:** the memory pipeline redacts secret patterns before writing
|
||||
memory artifacts; the dreamer must do the same *before* anything leaves the
|
||||
machine as Jev state, and re-redact before synthesis.
|
||||
- **Overlap with memory:** memory consolidation produces knowledge summaries;
|
||||
dreaming produces config changes. They share S0. If `memory.backend: local`
|
||||
is enabled, reuse its per-session extraction cadence markers instead of
|
||||
re-parsing unchanged sessions (the pipeline tracks processed-since mtime).
|
||||
- **Where outputs land:** skills into `~/.omp/agent/skills/<name>/` (native
|
||||
provider, priority 100; beats same-named lower providers), rules per the
|
||||
RULES.md contract (short, sticky, native locations only), tools into
|
||||
`~/.omp/agent/tools/`.
|
||||
|
||||
## Cost model
|
||||
|
||||
Per dream run (rough, at self-reported prices):
|
||||
|
||||
| Stage | Volume | Cost |
|
||||
| --- | --- | --- |
|
||||
| S2 Jev: 300 sessions → ~1,200 segments x ~4k tokens | ~5 MTok input | **~$0.21** |
|
||||
| S4 LLM synthesis: ~5 clusters x ~15k tokens | ~75k tok | cents–$1 |
|
||||
| S0-S3, S6 code + SQLite | — | negligible |
|
||||
|
||||
A full-history dream costs well under $1. The LLM synthesis spend is the
|
||||
larger line item and scales with proposal count, which S3 caps.
|
||||
|
||||
## Safety: this is self-modification, treat it like it
|
||||
|
||||
1. **Human gate.** No direct writes to live config. PRs + proposals only; the
|
||||
user merges. (Also required by the house rule: no commits to main.)
|
||||
2. **Blast-radius cap.** ≤5 proposals per run, each with evidence links; a
|
||||
proposal without a session citation is rejected by construction.
|
||||
3. **Injection.** Sessions contain untrusted content (web pages, issue text)
|
||||
and "state is not hostile" applies — content can argue for its own
|
||||
improvement. Mitigations: redaction before Jev sees it, the LLM synthesizer
|
||||
independently reviews evidence, and the human gate sees the evidence too.
|
||||
4. **Feedback loops.** The dreamer must not grade its own artifacts as
|
||||
"accepted" — outcomes come from the human's merge/reject actions only.
|
||||
5. **Reversibility.** Config lives in git; every accepted proposal is a
|
||||
reviewable commit; `RULES.md` stays short so the sticky layer can't bloat.
|
||||
|
||||
## Implementation notes
|
||||
|
||||
- Adapter interface first: `JevClient` with three backends — `jev` (real API,
|
||||
needs `TYPESAFE_API_KEY` early access, or Vercel AI gateway), `llm` (a
|
||||
frontier LLM constrained to the same schema, e.g. via TypeSafe's published
|
||||
`system-one-adapter-python` shim), and `mock` (deterministic, for tests).
|
||||
The interface is the design; the backend is swappable, so the pipeline can
|
||||
be built and audited before the key exists.
|
||||
- Pin `jev-1.13.0` in config, log the `model` field from every response.
|
||||
- Dry-run mode writes an audit report (what would have been proposed, with
|
||||
Jev's raw probabilities) and nothing else.
|
||||
|
||||
## Rollout
|
||||
|
||||
| Milestone | Gate |
|
||||
| --- | --- |
|
||||
| M0: parser + S2 dry run over last 30 days, human reads the audit report | report is sensibly targeted (eyeball) |
|
||||
| M1: full pipeline, thresholds strict (`config_gap > 0.8`), ≤3 proposals/run | user accepts ≥1 real artifact |
|
||||
| M2: weekly cadence, thresholds tuned from calibration data | calibration curve within ±15pp of diagonal |
|
||||
|
||||
## Open questions
|
||||
|
||||
- Does Jev's calibration hold on *this* corpus (session transcripts are
|
||||
long, messy, and self-authored)? M0 exists to answer this before anything
|
||||
acts.
|
||||
- Cluster-level evidence may need a second Jev pass with the current config in
|
||||
state (serial call, justified dependency per the primitives' independence
|
||||
rule).
|
||||
- Whether S4 synthesis should also run the doc-02 "config-drift guard" on its
|
||||
own drafts before proposing them.
|
||||
Loading…
x
Reference in New Issue
Block a user