PorkiCoder Research · Tab titles
Four beams, no new weights
The decisive gain came from search, not new weights: compare complete, safe pairs of words already on the page.
Abstract
Our previous result stopped 0.15 points behind the FLAN-plus-centroid benchmark. We kept the 35M B-9500 checkpoint and frozen centroid exactly as they were, restricted the namer to complete pairs of visible and representable page words, and let four beams rank those pairs under the model’s own logits. In fresh clean-development packets the resulting stack beat our title-tuned FLAN stack by 0.61–0.64 points. The frozen one-shot result then reproduced the relative win on the sealed 1,000: 6.20 versus 5.45, paired +0.752 (95% bootstrap interval 0.611–0.891). A strict 32-character variant led by 0.733. The preregistered final gate nevertheless did not clear: an identity-blind Codex audit rated 19/40 outputs useful (47.5%), short of the required 60%, even though the same audit favored ours over FLAN 18/9/13. Beam-4 beat the corrected greedy decoder by 0.45–0.48, while every continuation checkpoint we scored under beam-4—including 500 CE-only steps at 5e-6—underperformed unchanged B-9500. No parameters or learned components were added.
Final gate: not cleared
The sealed relative result is strong, but the gate was conjunctive. Four of five criteria passed; the independent absolute-usefulness criterion did not. This paper therefore records a successful sealed replication of the FLAN comparison, not a passed final promotion gate.
The sealed one-shot result
We froze the checkpoint, decoder, centroid, glue, four-way judge packet, audit sample, and analysis before generation. The sealed file matched its preregistered SHA-256, and the complete clean-development pipeline replayed byte-for-byte before the final run. Flash-Lite saw the locked and bounded versions of our system and FLAN in one identity-blinded packet; no reference titles were shown.
| Sealed comparison | 35M beam-4 + v2 | FLAN + v2 | Paired delta (95% CI) | W / T / L | 35M ≥5 / ≥6 | FLAN ≥5 / ≥6 |
|---|---|---|---|---|---|---|
| Locked Hybrid A | 6.20 | 5.45 | +0.752 (0.611–0.891) | 603 / 67 / 330 | 74.9% / 63.5% | 60.6% / 44.6% |
| Bounded 32-character glue | 6.15 | 5.42 | +0.733 (0.593–0.873) | 601 / 68 / 331 | 74.1% / 62.3% | 60.3% / 44.1% |
Intervals are deterministic percentile intervals from 100,000 paired row resamples. The locked mean itself was 6.197 (95% bootstrap interval 6.081–6.312). Judge transport completed 1,000/1,000 rows with zero failures.
The independent audit and the preregistered decision
A fixed 40-row sample was scored from a reference-free packet whose per-row candidate identities were hidden until all judgments were written. On locked glue, ours was useful on 19/40 rows versus FLAN on 13/40 and won/tied/lost 18/9/13. The relative criterion passed, but the preregistered absolute bar was 24/40 useful. The observed 47.5% usefulness rate has a wide Wilson 95% interval of 32.9–62.5%; that uncertainty does not change the point-estimate decision rule fixed in advance.
| Final-gate criterion | Required | Observed | Decision |
|---|---|---|---|
| Blind Flash-Lite mean | ≥5.50 | 6.20 | Pass |
| Codex-audit useful outputs | ≥60% | 47.5% (19/40) | Fail |
| Codex audit wins + ties vs FLAN | ≥30% | 67.5% (27/40) | Pass |
| Empty rate | ≤1% | 0% | Pass |
| Top-title mass | ≤10% | 0.2% | Pass |
| Conjunctive final gate | All five | Four of five | Not cleared |
The final set is now consumed. It must not become another development set, and there is no tune-and-retest claim after this result.
This is a new result, not a revised table
The last paper said the 35M was still short of FLAN plus centroid and warned that beam search was too expensive to ship. The quality conclusion changed; the latency objection narrowed in a controlled Ada diagnostic, but remains open in the application. The winning change was comparing complete two-word titles instead of committing to the first locally likely token. That reverses the quality result, changes the research pin, and gives us a mechanism worth testing elsewhere.
“FLAN” here means the exact title-tuned FLAN-T5-small descendant used throughout this campaign, not untouched stock FLAN and not a claim that a weekend run reproduced Google’s instruction training.
The clean-development result that froze the system
Absolute means move when Flash-Lite sees a different set of anonymous candidates. Compare systems only inside one row below. The first row is a focused three-way packet; the last two comparisons share a separate six-way packet. Both used the reusable 1,000-row clean-development set to freeze the candidate before the sealed run above. Means are on a 0–10 scale, and W/T/L is always 35M versus FLAN. These rows explain selection; they are not substituted for the final result.
| Packet | 35M beam-4 + v2 | FLAN + v2 | Paired delta | W / T / L | 35M ≥5 / ≥6 | FLAN ≥5 / ≥6 |
|---|---|---|---|---|---|---|
| Focused three-way packet | 6.16 | 5.52 | +0.644 | 601 / 54 / 345 | 75.3% / 62.5% | 60.8% / 45.9% |
| Fresh six-way packet | 6.34 | 5.73 | +0.613 | 607 / 63 / 330 | 78.4% / 65.2% | 68.5% / 47.9% |
| Bounded 32-character glue | 6.30 | 5.70 | +0.601 | 605 / 61 / 334 | 77.8% / 64.2% | 68.2% / 47.1% |
The fresh six-way packet also included the pre-correction beam output, corrected greedy, and bounded variants. The occurrence-level safety correction cost 0.004 points in that same packet; the win was not caused by the bug it removed.
The one change
Greedy and beam-4 see the same legal titles. Greedy chooses the best next token, then lives with that choice. Beam-4 preserves four partial paths long enough to compare whole pairs. In two fresh packets, beam-4 beat corrected greedy by 0.481 (447/325/228) and 0.453 (472/321/207). Beam-2 helped but trailed beam-4 by 0.188; beam-8 was effectively tied at -0.015 and cost more. The default length penalty of 1.0 won its sweep.
We are writing an app that sends email invitations. The template looks bad in Outlook 2007 because Microsoft uses Word as the rendering engine.Greedy + centroid: ms not 2007 (2.1)
FLAN + centroid: Email Invitations 2007 (7.6)
Beam-4 + centroid: Microsoft Outlook 2007 (8.5)
A developer moving from SQL Server to a LAMP hobby project asks how to start using stored procedures with MySQL and PHP5.Greedy + centroid: ASP Into Reasons (2.0)
FLAN + centroid: SQL Server Reasons (4.5)
Beam-4 + centroid: Stored Procedures Reasons (8.0)
A developer wants a tested PHP library for safer picture uploads rather than designing one from scratch.Greedy + centroid: PHP Into Uploading (3.0)
FLAN + centroid: Upload Uploading Myself (2.0)
Beam-4 + centroid: PHP File Uploading (8.5)
“On the page” had to mean what the encoder saw
The first constrained decoder extracted words by decoding a truncated token sequence.
Removing punctuation and unknown pieces could merge fragments into words that never appeared, such as a regex becoming dd.
We moved lexical extraction back to the untouched source text and used fast-tokenizer offsets only to enforce the truncation boundary.
A second review found a subtler hole. A word could tokenize cleanly in isolation while its actual source occurrence overlapped a tokenizer piece missing from the reduced vocabulary. Eligibility now requires at least one complete visible occurrence whose overlapping source token IDs are all mapped and non-UNK; the offered target spelling must also be fully representable. This changed five of 1,000 raw titles.
Independent contract audit
All 1,000 corrected clean-development raw outputs contained exactly two distinct page words, each backed by a fully mapped occurrence inside the true 160-token prefix. Every target was at most eight tokens; maximum raw length was 31 characters. The sealed 1,000 repeated the full audit with a maximum of 32 characters. Violations in either set: zero.
We cleaned the training data. Training still lost.
The obvious next move was to continue B-9500 on exactly the two title-derived page words it should own, with the centroid attached only after inference. The rule was one immutable JSONL miner, one authenticated shared token pack, then cache-only trainers. That workflow exposed two data bugs before it exposed the larger modeling result.
| Corpus | Train rows | What the audit found |
|---|---|---|
| v4: full-body match | 1,277,706 | 252,280/1,287,706 train+validation rows (19.6%) selected a word beyond the 160-token prefix. |
| v5: visible prefix | 1,025,426 | 233,723/1,035,426 train+validation rows (22.6%) lacked a fully mapped visible occurrence. |
| v6: visible + mapped | 794,034 | 801,703 train+validation rows passed the strict schema-3 audit with zero violations. |
Cleaner targets improved literal copying and sometimes the greedy result. They did not preserve the ranking over complete pairs. The best v5 continuation was 0.647 behind B-9500 under beam-4. V6 at step 1,000 was 0.534 behind; at step 3,000 it was 0.571 behind even though its greedy decode edged B-9500 by 0.028 in that packet.
We then removed the auxiliary losses and made the update tiny. After only 500 CE-only steps at 5e-6, the beam result was already 0.601 behind the pin. At 2e-5 it was 0.758 behind. The pattern suggests that useful ranking was present in B-9500’s sequence distribution; optimizing short labels made that distribution easier to copy from greedily and worse to search.
The frozen centroid won its own ablation
We also tried more conservative centroid_v2 selection: a capped-picker bundle (minimum document frequency, capped IDF, and digit/hash filters), mapped-word-only filtering, and both together. Each tested cleanup reduced the product score. The frozen final candidate therefore uses the original centroid_v2 and locked Hybrid A ordering. No centroid text was attached during training.
The 32-character contract is a separate switch
Raw beam titles were already safe (maximum 31 characters), but locked Hybrid A can append a long centroid word. It exceeded 32 characters on 49 of 1,000 beam rows; FLAN’s locked glue did so on 73. A separate bounded combiner skips whole words that would overflow instead of slicing through one. That strict variant produced zero over-length titles and still beat bounded FLAN by 0.601. On the sealed set, locked glue exceeded 32 characters on 46 rows for ours and 50 for FLAN; both bounded candidates had zero violations, and ours retained a +0.733 paired lead.
Size stayed fixed; search overhead was modest in the Ada diagnostic
The namer remains 35,045,888 parameters, versus roughly 77M for FLAN-T5-small. No second model, reranker, or learned head was added. On identical RTX 6000 Ada boxes, nine fresh 100-row batch-one processes per decoder averaged 18.72±0.11 seconds for greedy and 20.33±0.21 for beam-4 (mean ± sample SD, n=9), an 8.6% increase. One 1,000-row batch-64 run took 18.19 and 19.82 seconds, a 9.0% increase.
Those timings include model load and candidate construction. They are controlled throughput diagnostics, not per-request production latency. Apple MPS showed a larger batch overhead. The beam path still needs measurement in the actual worker and on target hardware.
The frozen candidate and the decision
Relative win, final gate not cleared
B-9500 step 9500 → corrected source-word beam-4, max eight tokens → original frozen centroid_v2 → bounded Hybrid A wherever 32 characters is a hard contract.
The frozen stack beat FLAN relatively on clean development, on sealed Flash-Lite scoring, and in the small independent paired audit. That is a robust comparative result. It did not meet the preregistered 60% absolute-usefulness audit bar, so this study does not authorize a final promotion claim. The checkpoint and decoder remain useful shadow candidates; further product decisions need evidence collected without tuning on the consumed final set.
What we are not claiming
- We are not claiming the preregistered final gate passed. It completed at four of five criteria; independent-audit usefulness was 47.5% against a 60% requirement.
- The n=40 audit is imprecise: its usefulness interval crosses 60%, but the preregistered decision used the observed rate, not a post-hoc reinterpretation.
- Packet means are not interchangeable. The paired delta and W/T/L inside each packet are the evidence.
- The benchmark is our title-tuned FLAN descendant, not untouched
google/flan-t5-small. - The decoder can only choose page words. The frozen centroid still fills the remaining slot(s) and sometimes supplies the mistake.
- Ada and MPS process timings are not production request latency, energy, or memory measurements.
- The 32-character-safe result uses a separately scored bounded combiner; we did not silently mutate the locked baseline.
Reproduce and inspect the evidence
The paper is accompanied by a self-contained source and evidence index, including the exact decoder and glue source, frozen tasks and outputs, blinded audit materials, final score rows, environment pins, and SHA-256 manifest.
Readers do not need an unlinked copy of this project’s scripts/ directory to inspect the implementation or recompute the reported summaries.
Regenerating model outputs still requires the separately hashed B-9500 checkpoint; the evidence bundle makes that boundary explicit.
python scripts/generate_constrained_titles.py \ --model /path/to/B-9500-step-9500 \ --tasks evidence/tasks_final1000.jsonl \ --tokenizer assets/tokenizer \ --vocab-map assets/technical_250m_vocab.json \ --constraint-mode source-words --strip-wrapper \ --max-content-words 2 --max-source-length 160 \ --max-new-tokens 8 --num-beams 4 --length-penalty 1 \ --output reproduced_final_beam4.jsonl
Generic-title banning stays off; it hurt the product packet. The default source-word cap is 96.
Final records: frozen protocol, gate summary with intervals, contract audit, and identity-blind audit manifest.
Reproducibility hashes
- B-9500
model.safetensors:0f45a1fef0b549bac512243df0aadad64aedec2a6ade7c17af730247b0b3d52a - B-9500
config.json:bd146261ee5eaab464002123f5680cc17a0547daee9397fc2d54ec8e8870b8d9 - Frozen centroid_v2 JSONL:
a28aa740572bc369150006bb9e3a850e967555289046c480d64216f9390a160c - Corrected raw beam JSONL:
f736d9c1e232d99fc56d21bc208004c998c576e0aea085d2fd7742619dd90477 - Locked-glue JSONL:
5f95a123491fd1c8f9216ed149db57ac62f9bd8638ec0b564e2aa9b07cd471ca - Bounded-32 glue JSONL:
9c9b1bff3a5241e5442f0e0b07103d7472c67f6c5fc6f3b5168726ac0f069928 - Sealed final tasks:
4694f1e4b5269b525fd8fb595f45b9ce65d5f831fdd981537da07479831b6874 - Frozen final protocol:
238e1df1ba77e7583d0e9f032580a3bb85705f08cd2656d0c307de32c55c3789 - Sealed raw beam JSONL:
afc3198b622d25c6cc800c0365880c010bd67299bf887d30025a0409c1efc125 - Sealed locked-glue JSONL:
d28b3c41a596e9006b6d49f17ba8f9f1a3cc31e9cde1cc537c478681a1acd2c0 - Sealed bounded-32 glue JSONL:
8afa62396b8fa47a3cd4a8f54ecc4f8acb606ed142c65f4ca1745d0dbfd46118
Frozen implementation record: commit 931e57b. The final protocol was hashed before generation. Full campaign findings, packet paths, cache hashes, final-gate summary, and teardown record are mirrored in the adjacent reference directory.