SHLEX · 15 SEPTEMBER 2026 UTC

jeremiah_v1.
Four arms, twelve checkpoints.

The question

The previous continuation reached 240/240 synthetic transfer outcomes while older retention fell from 145/160 to 139/160. This experiment tests whether data composition, a lower learning rate or an earlier checkpoint can preserve both.

Matched experiment

All four full-SFT arms use Qwen2.5-Coder-1.5B-Instruct and start from the same September 9 incumbent with fresh optimizer state. Arms A/B use M0 replay; C/D use M1, with 8,000 new validated examples and 11,408 replay rows. The 8,000 new rows cover 7,290 unique semantic specifications, capped at four rows per specification. Both mixtures contain 19,408 rows. A/C use learning rate 1e-5; B/D use 3e-6. Each runs one epoch with microbatch 4, accumulation 4, seed 42, seeded shuffle, bf16 and maximum sequence length 1,792. Checkpoints at steps 303, 607 and 1,213 represent matched exposure.

One RTX 6000 Ada generated all diagnostics greedily. Incumbent P0 uses the current app context; incumbent P1 isolates the explicit environment-format context correction. The September 10 comparator and all candidates use P1. Commands execute through the Mac broker with independent fixtures and unchanged sandbox policy. Successful-empty tests also require a positive companion fixture.

Diagnostic results

Each cell shows executed outcomes · plan correctness. The 240 transfer, 160 retention, 280 wider and 600 development cases are selection diagnostics. They are not 1,280 independent acceptance cases.

What changed under matched conditions

These are changes in executed outcomes at the final checkpoint. Positive numbers mean more passing cases. Lower-LR comparisons move from 1e-5 to 3e-6; mixture comparisons move from M0 to M1. Context-only compares incumbent P0 with P1. Each training arm has one seed, so these differences do not establish stability across repeated training runs.

At the final checkpoint, lowering the learning rate from 1e-5 to 3e-6 changes M0 from 1009 to 836 outcomes and M1 from 1183 to 1023 outcomes, out of 1,280. This is a matched one-epoch comparison. It does not establish how the lower rate would perform with more training.

Selection and retention tradeoffs

The operator selected C final as the default jeremiah_v1, with D replacing it only if a D checkpoint strictly exceeds its total executed outcomes over the same 1,280 cases. A tie keeps C final. Better D checkpoints rank by total outcomes, development outcomes, total plans and then earlier step. Incumbents were evaluated on all four boards, including the new 600-case development board. The case counts define the weighting; they do not represent measured production traffic.

This choice was made after viewing A/B/C results. The original frozen rule required zero historical outcome or plan regressions and at least 38/40 passing both measures in each transfer family. Its results remain below as a separate audit. They are not the operator selection rule, and historical regressions are not erased by an overall improvement.

C middle scores 1183/1280 and C final 1183/1280. Their retention scores are 139/160 and 132/160; development scores are 572/600 and 576/600. The operator explicitly preferred C final as the default. Equal totals do not mean identical capabilities.

Selected model: paired outcomes against incumbent P0

These counts compare the same requests case by case. Improvements are previously failed requests now passing; regressions are previously passing requests now failing. Aggregate gains do not cancel the practical cost of those regressions.

Independent acceptance and limits

Acceptance opened: True. The selected nominee receives a paired historical MPS recheck followed by the sealed 600-case synthetic comparison. Both the original zero-regression rule and a historical check of positive paired gain plus 38/40 per transfer family are reported. Under the operator overall-performance policy, historical shortfalls are tradeoffs and do not withhold this single-nominee acceptance assessment. Synthetic acceptance requires 540/600 overall, at least 85/100 per stratum, positive paired gain and zero false acceptance on unsupported requests. The final C/D choice was locked before opening these 600 cases. Only that selected checkpoint and the two incumbent contexts were evaluated on acceptance; no second nominee is allowed. Independent human evaluation has not been collected.

historical on MPS

Selected model against incumbent P0: 212 improved outcomes and 32 regressions. Paired gain +26.47 percentage points; cluster-bootstrap 95% interval [+21.65, +31.79] points (451 clusters, 5,000 draws, seed 42).

filename-retention on MPS

Selected model against incumbent P0: 0 improved outcomes and 2 regressions. Paired gain -5.00 percentage points; cluster-bootstrap 95% interval [-12.50, +0.00] points (40 clusters, 5,000 draws, seed 42).

These 40 cases are already part of the historical wider board; they are not additional independent evidence.

acceptance on MPS

Selected model against incumbent P0: 312 improved outcomes and 4 regressions. Paired gain +51.33 percentage points; cluster-bootstrap 95% interval [+46.50, +56.00] points (300 clusters, 5,000 draws, seed 42).

Acceptance strata

• Synthetic language uses the same teacher family for authoring and review.
• Forty filename compatibility cases are already included in the wider board.
• Historical and development boards are selection diagnostics, not independent acceptance.
• Independent human requests, application smoke checks and production latency remain outstanding.
• One training seed per arm; matched differences are not estimates of variability across retraining seeds.
• Known corpora and boards were audited for exact overlap and conflicting labels; this does not establish semantic independence.
• Overall ranking weights each of the 1,280 cases equally: 600 development, 240 transfer, 160 retention and 280 wider. This benchmark mix does not estimate real-world request frequency.
• The operator changed selection after viewing A/B/C diagnostics: C final is the default; D must strictly exceed its overall outcome count. This is a disclosed post-hoc research choice, not a pass of the original frozen gate.

Next steps

Use development diagnostics to design the next intervention. Retire the opened acceptance set from independent testing and author a fresh sealed set before the next experiment.

Credentials is the first follow-up priority: 15 of its 17 failed acceptance cases involve a changed ordering field. Use contrastive examples that preserve requested ordering while varying provider and credential wording, and separately exercise limit, all-results and file-type constraints. Field counts can overlap; this is an observed output-error pattern, not a claim about the model's internal reasoning.

For a subsequent training experiment, prioritize the measured regression patterns below. Collect contrastive examples that change one requested field at a time, such as file type, output mode or scope, and pair each with retention replay. Keep the selected checkpoint fixed as a baseline and test one intervention at a time at matched exposure. Treat the opened acceptance set as diagnostic evidence from now on. Freeze fresh development and acceptance data before comparing additional variants, and keep the new acceptance set sealed while tuning on development.

Field errors among selected-model outcome regressions against incumbent P0 (a case can contribute to multiple fields): {"all_omitted": 1, "invalid_document_plan": 6, "scope_changed": 2, "types_invented": 11, "types_omitted": 11, "wrong_output_mode": 7}.

The requested model name is jeremiah_v1; its status is research incumbent selected. The research-model selection does not deploy the model into the installed application. Mac replay ran concurrently with the fixed GPU queue. All twelve full checkpoint files were recovered to the Mini with recorded SHA-256 verification before GPU deletion. Laptop copies were verified before final aggregation. Estimated Ada compute cost: $7.43 (controller estimate; final provider billing may differ). Public evidence includes aggregates and hashes only; corpora, private prompts, raw predictions, fixtures and weights remain private.