120 lines
6.7 KiB
Markdown
120 lines
6.7 KiB
Markdown
# Jev in the pi (Oh My Pi) harness
|
|
|
|
Use cases for a decision model inside a coding-agent harness. Every claim below
|
|
is grounded in the harness's own docs (`omp://`); Jev facts come from
|
|
[01-what-is-jev.md](./01-what-is-jev.md). The mental filter for each idea: Jev
|
|
answers *bounded* questions ("is this X?", "which bucket?", "how severe?") in
|
|
~100-500 ms for ~$0.0002. The harness's LLM does the thinking and the writing.
|
|
|
|
## Where the harness can host a decision model
|
|
|
|
| Surface | Mechanism | File location |
|
|
| --- | --- | --- |
|
|
| Hooks / extensions | event bus with `tool_call`, `tool_result`, `context`, `before_agent_start`, `session_*` handlers | `<cwd>/.omp/hooks/pre/*.ts`, `~/.omp/agent/hooks/pre/*.ts` |
|
|
| Custom tools | model-callable functions, Zod/arktype schemas | `~/.omp/agent/tools`, `.omp/tools` |
|
|
| Skills | `SKILL.md` capability packs | `~/.omp/agent/skills/<name>/` (also `managed-skills` via `autolearn.enabled`) |
|
|
| Context files | `AGENTS.md` (user + project), `RULES.md` (sticky) | `~/.omp/agent/AGENTS.md`, `.omp/AGENTS.md` |
|
|
| System prompt | `SYSTEM.md` (replace), `APPEND_SYSTEM.md` (add), `PERSONALITY.md` | `~/.omp/agent/...` |
|
|
| Memory pipeline | two-phase extraction+consolidation over past sessions | `memory.backend: local`, phases run on `default` / `smol` model roles |
|
|
| Raw material | persisted sessions (JSONL), prompt history (SQLite FTS5) | `~/.omp/agent/sessions/.../*.jsonl`, `~/.omp/agent/history.db` |
|
|
| Model routing | `models.yml`, roles (`smol`, `tiny`, `task`, `advisor`, ...) | `~/.omp/agent/models.yml` |
|
|
|
|
Jev is **not** a model provider in `models.yml` — it cannot chat, stream, or
|
|
hold a conversation, so it cannot be a session model. It plugs in through
|
|
hooks, custom tools, and batch jobs (see
|
|
[03-dreaming-with-jev.md](./03-dreaming-with-jev.md)).
|
|
|
|
## Use cases, ranked by fit
|
|
|
|
### 1. Pre-flight tool guardrail (`tool_call` hook)
|
|
|
|
The documented hook example blocks `bash` commands containing `rm -rf` by
|
|
string match. Jev upgrades this to a semantic risk gate: state = the command +
|
|
cwd context; questions = Noul "destroys or exfiltrates data", Score "blast
|
|
radius", Choice over {allow, confirm, block}. Confidence-gate it: auto-allow
|
|
when safe > 0.9, ask the user in between, hard-block above a destruction
|
|
threshold. Same shape applies to `write`/`edit` paths (protected files) and to
|
|
`bash` chains that look innocent but end in a push.
|
|
|
|
Caveat: the model reads literally; keep the deterministic regex checks too —
|
|
Jev is the fuzzy layer on top, not the replacement.
|
|
|
|
### 2. Prompt-injection screen on tool results (`tool_result` hook)
|
|
|
|
Web pages, issue bodies, and arbitrary repo files can contain text engineered
|
|
to steer the agent. A `tool_result` handler can ask Jev: Noul "contains
|
|
instructions attempting to steer the agent or exfiltrate secrets", and if it
|
|
fires, annotate or redact the content before it enters context. Cost per
|
|
screening ~$0.0002 — cheap enough to run on every external read. Note the
|
|
docs' own warning cuts both ways: injection-shaped state can also move Jev's
|
|
answer, so treat the verdict as a tripwire for human-visible annotation, not
|
|
an autonomous veto.
|
|
|
|
### 3. Verification of agent output (the "verify everything" pattern)
|
|
|
|
An `agent_end` hook or a `verify` custom tool that scores the turn's outcome:
|
|
Noul per acceptance criterion ("the reported fix actually reproduces-and-tests
|
|
the changed path"), Score "evidence quality", Choice over
|
|
{complete, partial, off-track}. This is the System One workflow shape TypeSafe
|
|
built its evals around, and it plugs directly into the harness's verify-before-
|
|
yield discipline: an independent cheap judge over the transcript before the
|
|
turn is declared done.
|
|
|
|
### 4. Memory pipeline pre-filter (cheap Phase 1)
|
|
|
|
The local memory pipeline runs per-session extraction on the `default`-role
|
|
model for every changed session (cap: 64 sessions/startup, 4k-token inputs).
|
|
A Jev pre-pass over each session could ask: Noul "contains durable signal"
|
|
(decisions, constraints, resolved failures), Score "density of signal",
|
|
Noul "worth burning an LLM extraction on" — so the expensive extractor only
|
|
runs on sessions Jev rates. At 64 sessions x ~4k tokens the LLM pass is a few
|
|
cents; Jev's version is ~1% of that and ~100x faster, which is what makes
|
|
scanning *every* session every cycle affordable instead of capped.
|
|
|
|
### 5. Task routing / cascade (which agent, which model)
|
|
|
|
The `task` tool takes an `agent` choice (scout / task / reviewer /
|
|
security-reviewer) and `workpool` spreads items. A Jev front-end classifies
|
|
each item: Choice over agent types, Score "difficulty", Noul "read-only
|
|
research" — routing cheap items to scout, reserving expensive models. Same
|
|
cascade logic TypeSafe demos on support tickets, applied to subagent spawn.
|
|
This is advisory: a wrapper (custom tool or hook around `task`) makes the
|
|
call; the harness still owns control flow.
|
|
|
|
### 6. Config-drift guard on self-edits
|
|
|
|
When an agent writes to `AGENTS.md`, `RULES.md`, or a skill (the dreaming
|
|
pipeline in doc 03 does exactly this), a hook can gate the write: Noul "this
|
|
change encodes a one-off incident as a permanent rule", Noul "this rule
|
|
contradicts an existing rule in state". Cheap insurance against an agent
|
|
slowly cluttering its own configuration — the failure mode auto-learn systems
|
|
drift into.
|
|
|
|
### 7. Batch triage over history: the RSI dreaming loop
|
|
|
|
The biggest one gets its own document: replaying past sessions to mine
|
|
improvement opportunities (new skills, AGENTS.md/RULES.md edits, system-prompt
|
|
updates, new tools). Jev is what makes scanning *all* past interactions
|
|
economical. See [03-dreaming-with-jev.md](./03-dreaming-with-jev.md).
|
|
|
|
## Not Jev-shaped (do not force it)
|
|
|
|
- **Session titles, commit messages, summaries, compaction** — these are
|
|
*string generation*; the harness routes them to `tiny`/`smol` LLM roles for
|
|
good reason. Jev has no decoder; it cannot write them.
|
|
- **Code generation, refactors, reviews that need new text** — bounded-output
|
|
scoring can *rank* candidates but cannot produce them.
|
|
- **Anything needing knowledge outside the state you send** — Jev has no
|
|
world knowledge beyond state; no lookups. Retrieve first, judge second.
|
|
- **Image/screens understanding** — text-only state (as of Jev 1.13).
|
|
|
|
## Honest limits before betting on any of this
|
|
|
|
- Early access product; `jev-latest` currently resolves to `jev-1.13.0` and
|
|
moving GPU capacity means rate limits (250k tok/s, 1200 req/min) and
|
|
behavior can shift. Pin the version and log the response's `model` field.
|
|
- All performance numbers are TypeSafe's own, launch-week, unreproduced.
|
|
- Calibration ≠ correctness: validate every threshold against local outcomes
|
|
before letting it act autonomously.
|
|
- State ceiling: 64k tokens per call. Whole sessions don't fit; segment first
|
|
(doc 03 does). |