Bash SFT note 02: 3,000 executed examples, four arms, and a fair-base correction

21 August 2026 · cumulative Session 2 report · corrected 21 August 2026 · research note, not a paper · follows note 01

Session 1 taught us that lower imitation loss can coexist with zero real-task wins. Session 2 replaced “exit code 0 means verified” with synthetic fixtures, exact outputs, and filesystem snapshots. The original prompt explicitly required exactly one raw command with no prose or Markdown fence, but its strict 0/98 stock score still conflated adherence to that form with Bash semantics; it supplied no format demonstration and did not disclose the judge’s exact command-policy allowlist and restrictions. We corrected the comparison: with the same contract-calibrated prompt, stock passed 8/98 and Arm 2 passed 89/98. The raw-format confound is gone; the large in-domain gap and the failed external canary both remain.

Everything since Session 1

Do not merge the two Session 2 tests

The earlier 500-row result and the latest 98-task result answer different questions. Their percentages are not comparable.

*The original strict stock score is retained as an audit trail, not as the fair headline comparison. See the dated correction below.

The 137-row pilot, for completeness

The data gate changed

The project describes the 3,000 GFR candidates as trace-inspired synthetic examples, not copies of private sessions, but the public evidence has no source-hash or archetype-to-row mapping to audit that claim independently. We first distilled 16 execution-backed seed examples plus explicit generation constraints. The released prompts, paths, files, expected outputs, and decoys were synthetic and privacy-screened. Candidates stayed quarantined until a separate program could prove their results.

• Execution first: 3,000/3,000 final rows passed expected exit code, stdout, stderr, and before/after filesystem immutability on macOS Bash 3.2.
• Repairs disclosed: 10 rows had fixture or wording corrections applied before the recorded promotion execution. They were not silently relabeled.
• Grouped split: split groups connected by identical normalized commands or fixture setups stayed together. Exact normalized-command overlap among train, validation, and test was zero. We did not complete a semantic-near-duplicate or source-time leakage audit.
• Release: 2,402 train / 297 validation / 301 test rows across 998 connected components. The locked score uses one deterministic representative from each of 98 test components.
• Scope: 20 families across find, rg, jq, and sed. This is natural-language file search and retrieval, not all of Bash.

Four arms from the same pinned base

All arms started from the same stock Qwen/Qwen2.5-Coder-1.5B-Instruct revision. Training was response-only full-parameter bf16 SFT, batch 4, gradient accumulation 4, max length 1,024, 10 warmup steps, seed 42. The three full-data arms used 297 validation rows; the representative arm used 98. Generation was greedy.

Arm 2 minus Arm 1 is +4.1 percentage points; paired component-bootstrap 95% interval 0.0 to +9.2 pp (10,000 resamples). Two epochs is the best observed recipe, not a settled rule. Arm 4 changes row count and update count together, so its loss cannot isolate a causal “variants help” effect.

Correction: a fair base-model comparison

Why this correction exists. The original base prompt did state “exactly one raw command,” “no prose,” and “no Markdown fence.” Stock failed that explicit instruction. But the prompt gave no format demonstration and did not disclose the judge’s exact command-policy allowlist and restrictions, while Arm 2 had repeatedly seen the target response shape during SFT. Treating every format miss as a semantic miss therefore gave the trained model an avoidable calibration advantage.

Original strict score: retained, but no longer the headline comparison

*No command was executed for the gated models, so “unchanged” is vacuous. Stock wrapped all replies in fences or prose despite a system instruction requiring one raw command. Previous S2 produced 46 one-line replies, but its generated shell constructs failed the read-only policy.

Check 1: symmetric whole-response Markdown adaptation

We re-scored the original completions with the same deterministic adapter for every model. It accepts only a whole response that is already a single command or a single Markdown-fenced command; it does not select a promising snippet from prose or rewrite shell. The extracted command then faces the unchanged policy, syntax, execution, output, and immutability checks.

Arm 2 minus stock is +80.6 percentage points; paired component-bootstrap 95% interval +71.4 to +88.8 pp, 10,000 resamples with seed 42. The adapter fixes Markdown presentation, not command semantics.

Check 2: identical contract-calibrated generation

This is the stronger correction. Pinned stock and the pinned Arm 2 checkpoint received byte-identical effective prompts that restated the exact one-command output format and disclosed the evaluator’s allowed-command policy. The shared context included one neutral no-op user→assistant format demonstration whose assistant answer was true; it taught response shape without demonstrating a file-retrieval solution. Each model generated on a separate NVIDIA RTX 6000 Ada Generation GPU with greedy decoding and seed 42. The same judge then scored both. No adapter was needed.

Arm 2 minus stock is +82.7 percentage points; paired component-bootstrap 95% interval +74.5 to +89.8 pp, 10,000 resamples with seed 42. The two temporary generation droplets were deleted after the aggregate artifacts were pulled.

What the winner can and cannot do

Arm 2 passed 14 of 20 small held-out families perfectly. It produced 98/98 one-line, policy-safe, syntax-valid commands and made 0 mutations. Ten semantic failures remained: four exit-code mismatches and six stdout mismatches; no timeouts.

The post-selection canary

The locked historical task asks for handoff-like files under the current directory, prunes dependency/build/release trees, handles spaces, and returns a NUL-safe newest-first long listing. Its key vocabulary appears zero times in GFR train, validation, or locked test. Stock, previous S2, and all four GFR arms scored 0/1.

Arm 2 generated:

find 'current directory' \( -type d \( -name 'node_modules' -o -name 'dist' -o -name 'release-*' \) -prune \) -o -type f \( -iname '*handoff*' -o -iname '*hand-off*' -o -iname '*handover*' \) -print | sort -rn

It recovered all three name patterns and much of the pruning intent. It also literalized “current directory,” omitted NUL-safe filename handling, and sorted path text rather than long-listing files by modification time. That is exactly the compositional boundary the next training set should target.

What Session 2 taught us

1. Execute the outcome; do not bless the process

Exit status zero is not task verification. The old corpus often proved only that a command ran. The new promotion gate checks exact results and forbidden changes. This changed model selection from “looks like gold” to “returned the requested files.”

2. Split semantic relatives together

Random row splits leak paraphrases and fixture variants. GFR joins rows by declared split group, normalized command, and fixture setup before assigning components. That removes exact command crossover. It still does not remove generator-family similarity, so an external suite remains mandatory.

3. The response contract is learnable—and can confound a benchmark

The original strict run made stock look like a semantic 0/98 because it fenced every answer. Once we adapted Markdown symmetrically, stock passed 9/98. Once we disclosed the exact interface identically, stock emitted raw commands 98/98 and passed 8/98. Contract adherence still matters for direct execution, but it must be reported separately from command semantics.

4. More rows and more updates are confounded in our representative arm

The 802-row Arm 4 lost to the 2,402-row Arm 1 by 13.3 points (95% interval −21.4 to −6.1). Because Arm 4 also received one third as many examples and updates, this experiment does not prove which variants mattered. A causal repetition test should equalize optimizer steps or sampled tokens.

5. In-domain execution is necessary, not sufficient

89.8% on a frozen execution set is meaningful. The 0/1 external canary is also meaningful. The pair says “the pipeline can teach a narrow capability” and “we have not yet shown it transfers.” Publishing only either number would distort the result.

6. Exact command match is not the task

Arm 2 exactly reproduced 80 gold commands but passed 88 fixtures. Eight semantically correct alternatives would be marked wrong by string matching. Exact text remains a useful diagnostic; executable outcomes must select the model.

7. Freeze external canaries before generating repairs

The historical canary was reconstructed after arm selection, which makes it a useful alarm but only one data point. The next run needs 20 or more generator-independent tasks frozen first, covering deictic roots, compound constraints, safe filenames, metadata, and ordering.

8. Reproducibility includes operational failures

The winning 2.9 GB checkpoint was hashed locally and remotely before cleanup. All completed-run GPU droplets were deleted. The later four-arm repair attempt spent its remaining clock on transfer/provisioning and trained nothing; those four GPUs were also deleted. The fair comparison used two additional temporary GPUs, one per pinned model, and both were deleted after artifact pull. Prebuilt environments and an explicit stop-new-work cutoff would recover that time.

9. The evaluator is not yet a hostile-shell sandbox

Lexical rejection plus snapshots caught the read-only task class used here, but it should not be exposed to arbitrary model output. Add an AST-aware command policy or OS/container boundary before production evaluation.

The next path forward

1. Freeze at least 20 real-use, generator-independent execution tasks before creating new training rows. Keep the exact historical canary out of training.
2. Build unrelated repair examples for “here,” “this folder,” and “current repo,” plus commands that compose pruning, multiple patterns, spaces/NUL safety, metadata, and time ordering.
3. Compare a fresh stock-to-original-plus-repair arm with a low-learning-rate continuation from Arm 2. Use the external suite for selection and the 98-task GFR set only as a regression gate.
4. Retain all three audit views: original strict product behavior, symmetric whole-response adaptation, and identical contract-calibrated generation. Never present the strict 0/98 alone as stock semantics.
5. For a distil-lfm25-shellper comparison, reproduce its public benchmark contract separately. Tool-routing accuracy is not interchangeable with direct natural-language-to-Bash execution.
6. Equalize optimizer steps when testing whether repeated fixture/paraphrase variants add information.

Next success criterion: pass previously failed external tasks without losing the winner’s 100% raw-command, policy, syntax, and immutability rates—not merely raise the GFR aggregate.

Evidence and reproducibility

Public packet: ./bash-sft-02/. It contains aggregate metrics, the sanitized data manifest, recipe comparison, fair-evaluation summaries, canary summary, and hashes. It contains no private traces, prompts, model completions, fixtures, local or remote paths, machine addresses or identifiers, credentials, or weight tensors.

• summary.json — stock, prior S2, and all four arms
• arm-comparison.json — paired component-bootstrap recipe deltas
• fair-eval-summary.json — aggregate protocol and result counts for both corrections
• fair-eval-comparison.json — sanitized paired deltas and bootstrap intervals
• data-manifest.json — sanitized promotion counts and leakage checks
• canary-summary.json — aggregate external result and failure taxonomy
• report.md — compact machine-readable companion narrative
• SHA256SUMS

Winning checkpoint: Qwen2.5-Coder-1.5B-Instruct full SFT, 2,402 rows × 2 epochs, LR 2e-5. The unpublished model.safetensors SHA-256 is 6b1eb660ba974b292da07fe9c93359536abae2550044d6076cf645da2ed3b554. Training wall 113.7 s; peak GPU memory 15.22 GiB on one NVIDIA RTX 6000 Ada Generation GPU.

The experiment ran from a dirty, uncommitted source worktree. This packet freezes sanitized aggregate evidence, not the private full run directory; it should not be described as a hermetic reproduction bundle.

Bottom line

Session 1 found a broken measurement target. Session 2 built a much better one and produced a model that is genuinely strong inside it. We then found and corrected an unfair base-model interface disadvantage. The honest result is not “Bash solved,” nor is it “stock scored zero.” It is: stock passed 8–9% after fair interface treatment, Arm 2 passed about 90%, and transfer remains the next unsolved problem.