The Sniff Test

The average stood still. The model did not.

The sniff checkpoint scored 0.3512 against the teacher; the baseline scored 0.3519. Their paired mean difference was -0.00066, but they emitted different titles on 661 of 907 tasks. The apparent tie hides a redistribution of errors.

The sniff checkpoint won on 285 tasks by S, tied on 306, and lost on 316. Mean absolute movement was 0.0907, about 138 times larger than the net mean. The judge advantage was concentrated where S also rose: those rows summed to +244 judge points, versus -108 where S fell and +2 on S ties. This accounting does not identify why judge scores differed or rule out dimensions orthogonal to S.

Nor did the sniff rise on a lexical task-overlap proxy or distributional diversity. Unique titles fell from 313 to 306, title entropy fell from 5.149 to 5.101 nats, and mean task-token overlap fell from 0.2962 to 0.2924. Review Auth expanded from 16 to 51 outputs; on those 51 sniff rows, paired judge differences summed to -44. One narrower output pattern differed.

The sniff checkpoint stuttered less.

Without using judge scores in its definition, we post hoc flag a title that repeats any case-insensitive ASCII-alphanumeric token, as in Test Test or Bot Bot Status. The exploratory flag is a deterministic property of the generated title.

The same direction appears in one nested subset and one older ID-disjoint panel. On 288 tasks with known conversation threads disjoint from training, repeats fell from 46 to 30 (nominal exact paired p = 0.0166). On the repeatedly inspected 62-task panel they fell from 10 to 4 (p = 0.070). In the full set, repeated-token titles of at least three tokens fell from 121 to 70, a 42% reduction. The number of titles that long barely moved, from 430 to 426, arguing against an explanation based solely on fewer titles reaching three tokens. Exact two-token doubles changed only from 24 to 22 (p = 0.824). Removing repetition aligned with +65 judge points and introducing it with -29. Their +36 arithmetic difference is 26% of the total +138, not a causal mediation estimate.

A local decoding change can look like learned restraint.

Repetition is a known failure mode of autoregressive generation. One token wins, becomes the next decoder input, and helps itself or a related frequent token win again.

T5 ties its input embedding to its output projection, so the intervention touched both sides of this loop. A copied row could change how a token enters the decoder and how decoder state becomes its next-token score. It need not teach software semantics to alter local margins among common words.

This sketch explains how one checkpoint could change, not why the grid changed. KD pulls students toward one teacher, but training can preserve different boundaries near greedy ties. A small displacement can reroute decoding without moving the average. Prior work establishes the ingredients; we did not measure token margins inside these models.

This is underspecification: random training choices return models with similar aggregate scores but different stress-test behavior. The original fixed pair reduced repeated titles by 53, but it was confounded. One matched ordered arm reduced them by 44 against its body-matched baseline, yet direction varied across the grid. Preserving Verne's order improved neither repetition nor fidelity on average.

The favorable direction does not recur consistently.

Audit found that the original baseline and sniff checkpoint did not start from the same transformer body. The transplant script created a fresh, unseeded body in a separate process. The original comparison therefore changed both the embedding rows and the body initialization.

We recommissioned an RTX 6000 Ada and built a crossed follow-up. Three body seeds were held bit-identical across arms. Three donor seeds created each embedding treatment. Every arm received the same 3,000-step KD schedule. The designated baseline and no-op pair produced identical histories and outputs; its run log recorded identical checkpoint hashes before those weights were stripped.

The random donor is not a pure pretraining control: it was initialized and saved in fp32, whereas ordered and shuffled donors trained in bf16 before fp32 export, so random-versus-trained does not isolate pretraining from precision. Ordered versus shuffled shares initialization, precision path, token multiset, and schedule, making it the narrower sequence-order test.

Complete result, 31 of 31 runs. Ordered Verne averaged 0.96 percentage points, or 8.7 titles per 907 tasks, more repetition than shuffled Verne. Its teacher similarity averaged 0.00358 lower and improved in only four of nine matches. After subtracting each body's baseline and averaging the nine fixed cells, all three families had more repetition and lower S overall: random +1.67 points and -0.00773 S, shuffled +1.00 and -0.00291, ordered +1.96 and -0.00649. The fixed grid is too small for a population estimate and provides no support for the proposed stable sequence-order advantage.

A small experiment with a narrow claim.

Model and objective

The teacher is a 77M parameter FLAN-T5-small model fine-tuned for terminal-tab titles. The comparator, kd_vocab, is a randomly initialized T5 student with width 384, four encoder layers, two decoder layers, and a 6,985-id vocabulary. The sniff arm replaces its tied input and output rows with rows from a 32,100-id Verne donor. The original script did not match the remaining random body. Targets are valid teacher replies, not human-gold titles. Training minimizes:

The KL direction above matches the implementation. The sniff uses sequence length 512, 15% corruption, mean span length 3, batch 32, AdamW learning rate 2 × 10^-3, and 50 warmup steps. Its Python seed 42 controlled masking and order, but the original donor initialization was not torch-seeded. KD uses batch 32, learning rate 3 × 10^-4, 100 warmup steps, seed 42, and 3,000 steps. Decoding is greedy with a 12-token cap and no-repeat-bigram constraint. That constraint blocks a repeated bigram, not a unigram appearing twice. It does not enforce the desired one-to-three-word product rule. Exactly two words occurred on 460 of 907 sniff titles and 461 baseline titles.

Evaluation and uncertainty

The automatic score is the mean of normalized character similarity and token F1 against the teacher title:

Reported 95% intervals resample 20,000 evaluation-thread clusters; rows without a known thread are singleton clusters. Exact McNemar tests use discordant row pairs and are labeled nominal because they do not cluster threads or adjust for post hoc metric selection. Intervals describe this task collection, not retraining seeds or users. The judge figure also reports an unweighted t sensitivity across nine complete 100-task scoring blocks. Those sequential blocks came from one evaluator and are not independent replications. All 907 analyses are exploratory because this set was repeatedly inspected during the campaign.

Data boundary

The 907 rows are ID-disjoint from the title-training split, but 52 serialized inputs occur in training under other IDs. Of 758 rows with known conversation IDs, 470 share a thread with training; 288 are demonstrably thread-disjoint. On the full set, exact teacher-title counts are 62 for baseline and 63 for sniff. After removing exact-input overlaps, they reverse to 56 for baseline and 55 for sniff. The S tie (-0.00036) and repetition gap (130 to 82) remain. We therefore call this a reused development evaluation, not an untouched holdout.

What survived the sniff test.

Three decisive tests, in order.

1. Test the decoding mechanism. Compare tied and untied embeddings while tracking token logits and top-two margins. If the proposed feedback loop is real, margin changes should precede repeated decoding.
2. Predict robust starts instead of cherry-picking them. Screen many starts with inexpensive early diagnostics and a multi-objective validation rule, then evaluate the chosen checkpoint once on untouched data. Compare this seed-screening policy with random restarts under equal compute.
3. Test the size claim. Use identical precision paths, more body and donor draws, several model sizes and corpora, a locked thread-disjoint set, and blinded randomized-order human ratings.

1. Raffel, C. et al. 2020. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. JMLR 21(140).
2. Hinton, G., Vinyals, O., and Dean, J. 2015. Distilling the Knowledge in a Neural Network.
3. Gururangan, S. et al. 2020. Don't Stop Pretraining: Adapt Language Models to Domains and Tasks. ACL.
4. Press, O. and Wolf, L. 2017. Using the Output Embedding to Improve Language Models. EACL.
5. Holtzman, A. et al. 2020. The Curious Case of Neural Text Degeneration. ICLR.
6. Welleck, S. et al. 2020. Neural Text Generation with Unlikelihood Training. ICLR.
7. D'Amour, A. et al. 2022. Underspecification Presents Challenges for Credibility in Modern Machine Learning. JMLR 23(226).
8. Mosbach, M., Andriushchenko, M., and Klakow, D. 2021. On the Stability of Fine-tuning BERT. ICLR.
9. Reimers, N. and Gurevych, I. 2017. Reporting Score Distributions Makes a Difference. EMNLP.
10. Dodge, J. et al. 2020. Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stopping.