01
Fixed the refusal contract first
unsupported:<slug> is parsed before shell policy, displayed explicitly, and never executed. The unsafe old idea—using the allowed shell no-op :—is rejected.
This is a rejected release candidate, not a deployed model. The existing Shlex checkpoint remains in place. The candidate missed the predeclared human-use threshold and produced both a false refusal and a secret-sensitive search.
The old benchmark was real—and incomplete
The incumbent passes 92 of 98 frozen macOS execution tasks. That result is reproducible, but the tasks are long and specification-shaped. People using a desktop pet ask for files with fragments, shorthand, missing nouns, ambiguous scope, and ordinary conversational filler.
In-generator command skill
- Known prompt grammar and fixture conventions.
- Detailed requests that expose their own edge cases.
- Historical shell families under the original scorer.
Semantic usefulness
- Very short or elliptical natural language.
- Correct whole-folder, child-folder, type, name, topic, and time distinctions.
- An honest unsupported result when local filename tools cannot answer.
The first audit of 13 completed searches found only two useful results. Errors included turning filler into a directory, repeating the already-selected root as a child, broadening an exact dotfile request, losing a topic, and claiming success for questions that require looking inside an image.
One boundary changed; the safety boundary did not
Earlier product code tried to recover known meaning after generation. Live evidence showed that these “helpful” rewrites could also corrupt good commands. The model should learn user intent. Deterministic code should enforce what can be proven.
Learn the distinctions
- HTML files versus filenames containing
html. - The selected tree versus an actual named child.
- Filename versus content search.
- Creation time versus modification time.
- Answerable search versus unsupported request.
Keep the hard boundary
- Fixed root, relative paths, command allowlist, and blocked flags.
- macOS write denial and refusal to run without the sandbox.
- A non-shell unsupported outcome that is never executed.
- Trusted recency ordering, result limits, and output confinement.
No per-word patch: we did not add an HTML rewrite, widen xargs approval, allow find -exec, weaken the sandbox, or overwrite the historical benchmark.
What we changed
02
Kept evaluation separate
The old 98 stayed frozen. New data used new private paths, new fixtures, all sides of every contrast, and an additive product-policy scorer shared with the app.
03
Rebuilt prompt-label pairs from one intent
The failed v2 dataset had shortened prompts paired with commands that still referred to old words or roots. V3 renders both prompt and gold from the same structured intent, then executes every supported gold.
04
Trained matched continuations
The first faithful mix contained 5,424 rows. A 6,666-row remix replayed all 2,402 historical train rows, added more whole-tree practice, and balanced only the refusal families that failed.
05
Selected before opening the canary
The alternate-initialization arm was selected from frozen and synthetic results. Only then did it see the 13-query private canary once, under the production envelope with semantic rewrites disabled.
The training data
The final remix contained 6,666 private rows. It was designed around actual failure archetypes without copying exact user queries, personal paths, or product history into training.
2,402exact historical replay rows
4,264faithful conversational rows
65.19%of
find commands rooted at .560typed unsupported outcomes
80new execution-backed hard tests
0supported golds accepted without execution
Matched results
All generations were greedy and executed on macOS. The four historical newest-file items are report-only because their old gold uses a command shape the product now rejects.
| Arm | Start | Frozen v1 | Non-newest | Live hard | Supported | Unsupported |
|---|---|---|---|---|---|---|
| Incumbent control | last40-wd01 | 92/98 | 88/94 | 7/80 | 7/72 | 0/8 |
| Faithful v3 | last40-wd01 | 90/98 | 86/94 | 74/80 | 72/72 | 2/8 |
| Data remix | last40-wd01 | 90/98 | 86/94 | 79/80 | 71/72 | 8/8 |
| Alternate init | kimi-hardfam-wd01 | 91/98 | 87/94 | 79/80 | 71/72 | 8/8 |
The faithful rows fixed every supported synthetic case. Balancing the specific refusal failures fixed all eight unsupported cases. Both remixed models still confused an HTML file-type request with a filename-substring request.
Then the human canary failed
The selected candidate made every output syntactically valid and stayed inside the chosen folder. It still did not preserve enough meaning.
- Four semantically useful results, up from two.
- A correct refusal for a request needing image understanding.
- A useful bounded large-image search.
- Correct explicit child-folder plus HTML-type behavior.
- An exact dotfile-name search.
- One false unsupported result on an answerable topic search.
- Selected-root repetition and incorrect hour/day conversion.
- Creation and modification time confusion.
- Dropped provenance constraints.
- One unjustified dotfile-content guess, violating the secret-sensitive gate.
The important result is 4/13, not 79/80. A live-style prompt is not merely a short synthetic prompt. V3 still shared vocabulary, intent factorization, and command expectations between its builder and its test. It measured interpolation inside a better generator.
What counts as a benchmark now
Untouched product queries
A frozen private canary drawn from future real use. It judges usefulness, false refusal, secret-sensitive behavior, and root scope. It may veto shipping; it may not choose training arms.
Independent language plus history
New prompts and fixtures authored separately from the training builder test transfer. The old 98 remains a family-level regression report, not the product's final score.
The opened 13-query set is now a regression set. It cannot be the untouched proof for the next candidate. We can learn from its failure categories without copying its exact wording into training.
The next run
The next experiment changes one variable: data. The recipe and starting point stay fixed unless training itself becomes unstable.
Build 9,000–12,000 faithful rowsKeep the 2,402 historical replays. Add independently phrased core searches, compound hard pairs, refusal twins, and rehearsal for the families that regressed.
Broaden language, not just row countVary brevity, filler, omitted nouns, word order, punctuation, casing, singular/plural, and minor typos. Use new fixtures and multiple authors or independent generators.
Freeze before the gradientLock an independently authored hard set and a fresh private human canary. Never tune on either after opening results.
Keep one matched continuationOne epoch from the alternate initialization at 1e-5, cosine schedule, weight decay 0.1, batch 4, accumulation 4, seed 42.
Use product gatesRequire at least 95% on the independent hard set and at least 80% useful on the fresh human canary, with zero scope escape, false refusal, or secret-sensitive execution.
Ship behind rollbackOnly a passing model is deployable. Retire semantic rewrites individually after shadow evidence, starting with the rewrite that injected unrelated filename terms.
Reproducibility and privacy
All three complete 2.9 GB tensors were pulled locally, size-checked, and SHA-256 verified before the single Ada GPU was deleted. The protected droplets were untouched. Seventeen relevant unit tests passed, the training scripts passed shell syntax checks, and the data builder passed compilation and deterministic rebuild checks.
This report publishes aggregate scores and synthetic examples only. It contains no SFT corpus, private fixture, exact product query, personal path, raw prediction dump, or model tensor.