Shlex · local file-search model · 2026-09-17

Jeremiah v3: more rows for the weak families, trained on a rented RunPod GPU

What this run was for

Shlex is a desktop file-search pet. A 1.5-billion-parameter model on the person's Mac turns a plain request into one line that a sandboxed broker executes. Jeremiah v2 taught that model to search by what documents contain. It left some families weak, topic alternatives most of all, and a later reinforcement-learning pilot showed that reward cannot supply knowledge the training rows never held. So this run adds rows: 1,200 new reviewed requests for six families (many-worded topics, credentials named with no vendor, provider secret spellings, "in the X repo" scopes, document nouns such as runbook and changelog, and refusal contrasts), on top of the unchanged v2 mixture.

Mixture: 9,600 backbone rows + 2,400 v2 rows + 1,200 v3 rows = 13,200 rows, 825 optimizer steps of 16, snapshots at 206, 413, 825. Same recipe as v2 (full-parameter fine-tuning, learning rate 1e-5, cosine schedule, one pass) and the same start point, jeremiah_v1, so the v2 snapshots are a like-for-like comparison. Ten accepted rows were left out because their request text already appeared twice, and eight more so that every step holds exactly 16 rows.

How results are measured

Every number below is an executed test. The model's one-line answer is run through the real macOS broker on a fixture built from the case's expected plan, and the case passes only when the returned paths, or the refusal, match exactly. Generation is the app's own greedy decoding on Apple silicon. The trainer's token loss is logged and decides nothing. All six models were scored by the same script on the same boards on the same Mac; the v2 columns on the v2 boards are the published v2 receipts.

The v3 development board has 109 held-out cases. 13 of them turned out to share their request text with a training row: two families have no vendor or topic to hold out, and "files that mention api keys" can only be said so many ways. Those cases are shown on their own line and count toward no gate.

v3 development board

v2 development board

Operator canaries

Regression board

1,330 older cases: historical, incident and old acceptance boards. Scoring one model takes about 25 minutes, so v2 only scored its final snapshots here; the same holds for cells marked not scored.

Gates

The v2 gates as published, with the v3 board added: canaries 2/2; v2 content-first development at 90% and each of its six families at 80%; unseen v3 development at 90% and each v3 family at 80%; targeted development within five cases of the start point; regression within five cases of the model now in the app (v2 measured against the start point); zero false acceptances; zero unsafe executions.

Examples

Synthetic evaluation requests from the unseen v3 cases. Between the model in the app and v3 step 825: 26 cases fixed, 14 still failing, 5 newly failing.

Training on RunPod instead of DigitalOcean

Every earlier run used a DigitalOcean GPU droplet. For days the 48 GB cards there (RTX 6000 Ada, L40S, 1.57 USD/h) had no stock, which left the H100 at 4.41 USD/h. RunPod lists the RTX 6000 Ada at 0.84 USD/h on its Secure Cloud. The difference is what you rent: a Docker container on a partner's host with disk billed separately, not a whole virtual machine with everything bundled. Training needs a GPU, SSH and disk; it does not need a virtual machine. Private data stays on Secure Cloud rather than community hosts.

The controller kept its shape: a Mac mini at home holds the API key, creates the pod, uploads a hashed packet, watches the job, pulls every snapshot, verifies each hash, and only then deletes the pod. The GPU host never sees a credential, and a pod that has produced weights is never deleted before those weights are verified locally.

Five launches, one run

Every failed launch ended with the pod deleted by the controller and the deletion confirmed. Three of the four failures were my own mistakes in new code, and each could have been found without renting a GPU.

Where the 27 minutes went

The pod itself was reachable over SSH well under a minute after creation, faster than a droplet. Training ran at 0.77 steps per second with 16 GiB of the card's 48 in use, the same speed the Ada gave on DigitalOcean.

Friction log

Before the next run

• Starting weights from private object storage, not from home. Largest time saver.
• Our own container image with the pinned PyTorch stack baked in; the install step disappears and versions cannot drift.
• A balance check before creating anything: the account is prepaid, and a pod is stopped mid-run when it reaches zero.
• A GPU-free rehearsal of the packet, install and ssh choreography whenever that code changes.

Identities

Environment: NVIDIA RTX 6000 Ada Generation, driver 580.126.20, Python 3.11.11, torch 2.11.0+cu128, transformers 5.15.1.