SHLEX · 15 SEPTEMBER 2026 UTC

Jeremiah v2.
Find the file that holds the key.

The question

The installed Shlex model (jeremiah_v1) refused the operator's own request, find me files containing any gemini api keys, with unsupported:not-a-file-search, although the reader can execute the intended search perfectly (20/20 reader canaries, 44/44 wrong-plan mutants rejected). The gap was entirely translation: turning everyday words into the one-line plan the reader executes. This run asks whether a bounded supervised fine-tune from jeremiah_v1 closes that gap without damaging what the model already does.

Contract decisions frozen before any data was written

• Bare key mentions are content searches. "files containing any gemini api keys" is never refused and never silently switched to stored-credential detection.
• The model enumerates six spellings. space, underscore and hyphen forms, singular and plural, because the reader matches literally and case-insensitively.
• Key mentions include env files. .env is where keys live and is not a default type; stored-credential and quoted-identifier searches keep the established default types.
• Stored mode only on explicit wording. credential:true only for saved/stored/real/actual; provider name in all.
• Quoted identifiers are literal. "GEMINI_API_KEY" goes in all verbatim, no expansion.
• Listed topic wordings go in any. with singular and plural forms; unlisted synonyms are not invented.
• Category words are searches. "where are my invoices" is a content search, not a clarification.

Older rows that taught the opposite of these rules were excluded, never relabelled. The experimental v2 concept schema stayed isolated; every plan in this run is the installed v1 contract.

Loss is measured by tests passing

Token cross-entropy moved the weights and was logged as a diagnostic. Every number on this page is an executed test: the model's answer is run through the real macOS broker on a fixture built from the case's gold plan, and it passes only when the returned paths equal the expected set, order and truncation included, or when a refusal is the frozen slug. Nothing on this page is a loss value.

Data and gates

Two arms of 12,000 rows share a 9,600-row historical backbone and differ only in 2,400 intervention rows. Repair: 891 new content-first rows (frozen intents; Grok wording; wording checks; blind request-only review; 1264/1264 golds executed with 13,989 constraint mutants rejected) plus 1509 reviewed targeted-repair rows that survived the content-first audit and were re-executed today (3201/3216, 20,007 mutants). Control: 2,400 matched historical rows. Language: 1264 authored, 1163 accepted, 101 rejected and left rejected. Validation: 240 rows. Provider names are disjoint across train, validation, development and acceptance splits, so held-out scores measure generalisation to names never trained on.

Recipe, unchanged from the retention matrix: full-parameter response-only SFT, one pass, 750 optimizer steps (batch 16), learning rate 1e-5 cosine, snapshots at 188/375/750, seed 42, initialised from jeremiah_v1 (1a398f8880b23a18…), P1 context 529717c6555d6416…. An independent 30-worker Grok audit of the code, data and plan ran before training; its one high-severity data finding (an env-type inconsistency between new and reused rows) was fixed before the run.

Results

Canaries are the operator's two exact requests, held out of all training. Development mixes 295 reviewed targeted-repair cases with 108 new content-first cases on unseen providers. Regression holds the contract-consistent, fixture-runnable rows of the historical, incident and old-acceptance boards (113 legacy filename rows are scored by the earlier evaluator and 34 rows that contradict the clarified contract are reported separately, never counted).

Sample outputs

Synthetic evaluation requests only (plus the two operator canaries). A plan marked equivalent differs from the gold line but executes to exactly the same result on the fixture; that counts as correct. 1 of the 38 repair answers below are equivalent rather than exact.

Eligibility

No snapshot met every gate. Nothing is nominated or installed.

Cost and attempts

Limits

• One seed, one learning rate, one continuation; differences between arms are not estimates of variance across retrainings.
• All requests are synthetic (Grok-worded) except the two operator canaries; no human-authored acceptance set exists yet.
• A key stored only under a different name (for example GOOGLE_API_KEY for Gemini) is not found by either arm; unlisted synonyms are not invented; PDF contents remain unsupported.
• Same-model-family author and blind reviewer for the synthetic wording.
• Two regression requests for PowerPoint slide text were answered with a search carrying an invalid format type (pptx, ppt) instead of unsupported:needs-document-format. The reader rejects such a plan before anything runs, so the failure is inert, but it is counted as a false acceptance and listed here.
• The topic_alternatives family is scored by strict set equality on the plan; its misses are almost all plural-form mismatches, and several intended plurals in the frozen topic set ("crew trainings", "artwork lendings") are forms nobody would write. Treat that family's score as a defect of the intent set, not as evidence about topic retrieval; the next cycle replaces those plurals.

Identities

Frozen manifest 8f14960448d093f926396587d21d4a6cb796d8fae37493d72dde2b8c61ccd211 · intents f048a1278c0365cf2de008a03b00e4a759d242c29b198b9a38151611c4f840e5 · broker b02991dd682f10b651b22e85174ac5ad43e9b4256b9919aeda394ffe60418a9d.