While Not Good Yet

"While the counter is below ten, go round again." It is the oldest sentence in computing, and it works for a reason nobody notices any more: the machine can answer the question you ask it. Comparing a counter to ten costs one clock cycle and the answer is exact. The whole power of the loop sits in that cheap stopping condition.

The questions that actually decide whether a piece of work is any good offer no such courtesy. Is this chart readable? Does this paragraph follow from the one before it? Is this forecasting model good enough to put into production? None of them resolves in a clock cycle. So we took them out of our loops and moved them to the end, into a single pass, done by hand, once, by someone who has already done a lot of work that day.

The Ralph loop puts them back. It keeps the skeleton of the while loop and drops a judgement where the test used to be: a human eye, a vision model, a held-out set of data. The substitution looks modest. It changes what can be automated, and it opens one specific hole, which the rest of this article measures.

The loop in one line

The name comes from the Australian engineer Geoffrey Huntley, who described in July 2025 what he calls the Ralph technique¹, after Ralph Wiggum, the Simpsons schoolboy whose answers miss the point with admirable consistency. It fits on one shell line:

while :; do cat PROMPT.md | claude-code ; done

You write what you want into a file, feed it on repeat to a coding agent, meaning a language model allowed to read and write files and run commands, and you do not let it exit. The surprise is that this yields working software, and the reason is not the one you reach for first.

A language model works inside a context window, the amount of text it keeps in view during an exchange. That window is short, and its quality degrades as you fill it: late in a long conversation the model loses track of its own decisions. A code repository does not degrade. On every turn of Huntley's loop the agent starts from nothing, rereads the real state of the files, does one thing, and writes it down. Progress accumulates on disk and in version history, never in the model's memory. The forgetting is the feature.

Huntley does not sell the technique as a solution. He writes that it is "deterministically bad in an undeterministic world," that it only suits greenfield projects, that it needs a senior engineer steering, and that its output arrives "under baked, baked, or baked with unspecified latent behaviours." He reports a 50,000 dollar contract delivered for 297 dollars of compute; that figure is his, offered by the author of the method, and nothing in this article checks it.

What his command line leaves open is the whole subject here. In shell, while : means "while true." The condition is empty. The original Ralph loop has no test at all: it runs until a human stops it. Everything interesting starts when you put something back into that condition.

What replaces the test

Four moves describe every variant you will meet. Act: make one change, exactly one, so that later you know what to credit the result to. Render: turn that change into an artifact you can actually examine, an image, a printed text, a table of measurements. Judge: examine the artifact. Decide: keep it or drop it, writing down why.

The move most often skipped is the second. With nothing rendered, the judgement lands on an intention rather than on a result, and the loop turns without learning anything. The fourth is the one you regret skipping: with no reason written down, you will not know, three weeks later, how many ideas were tried before the one that survived. The last section shows that this count is not paperwork. It is a statistical quantity on which the validity of everything else depends.

Looking at what you just made

You proofread a sentence by reading it back. You cannot proofread a chart that way. The code that draws a figure can be entirely correct and the figure still wrong: a label sliding off the edge, a legend sitting on the data, two points landing exactly on each other, a red-and-green palette collapsing into one muddy colour for a reader who cannot separate them. None of that appears when you read the source. It appears only when you look at the result.

That discipline is what the Ralph Eyeball Loop² formalises, at sprezzature.ai: render, look, fix the source, go round again, until there is nothing left to catch. The tool takes four kinds of source, a web page, a TikZ figure, a Mermaid diagram, a hand-authored SVG, and turns each into an image with a deterministic renderer, no model and no chance involved: the same source always gives the same picture. It then opens a critique file named after a hash of the source's path, and appends a numbered section on every pass, so the file keeps the history of what was seen and fixed instead of overwriting it.

That critique follows eight prescribed headings: layout, contrast, visual hierarchy, spacing, accessibility, colours, whether every label can actually be read at the size it renders, and an overall verdict. A web page gets two more, what is visible without scrolling and what the layout does at that width. Two modes coexist: by default the agent that just wrote the code looks at its own output; with a flag, an offline vision model fills the form in first and you review it. The sprezzature gallery holds one hundred and twenty-seven kinds of chart, every one written directly in SVG and every one put through this loop, which is what gives the method its sample.

A concrete case says it better, and it comes from this very page. The articles on this blog descend from a layout tool's export. The article container there declares twenty-eight pixels of spacing between paragraphs, but the running-text style resets it to zero on each paragraph. The result: across all eight published articles, the blocks touched each other. The stylesheet was valid, the markup was valid, no test failed, no automated audit complained. It took photographing the rendered page and looking at it. The same pass turned up two more defects of the same family: a list rule that overrode the indent, leaving bibliography numbers sitting on the first letter of the note, and a container that preserved whitespace, so every newline typed between two list items became a blank line on screen. Three defects visible in a second of looking, invisible in the source.

Eyes you do not have

The loop earns the most where the judge sees what the actor cannot. Accessibility is the sharpest case. A person with ordinary colour vision cannot see their own figure the way a colour-blind reader sees it, and no amount of care changes that, because the information is not in their eye. A simulator redraws the image as those readers perceive it, and in plain grayscale. A review that is physically impossible becomes an ordinary step.

Grayscale is the severest single test. Complete colour blindness is rare, but a figure that reads with no colour at all reads under every milder deficiency. If the point survives grey, it survives everyone. The criterion has a second virtue: it is binary and settles in a second, which makes it nearly as cheap a stopping condition as comparing a counter to ten.

Rereading at the seams

The same skeleton applies to prose, and the writing charter this article follows makes a rule of it³. You do not reread a text paragraph by paragraph but by pairs of consecutive paragraphs, because flow does not live inside a block, it lives at the seam. You read paragraph n aloud followed by paragraph n+1, in one breath; you look at the move from one to the other; you fix the text itself, the order of the sentences, the last line of the first, the first line of the second, rather than noting in the margin that "it snags"; then you slide by one, onto the next pair.

The sliding is what does the work. The windows overlap, so every paragraph is tested twice, once as an arrival and once as a departure, and those are two different tests: a block can receive what precedes it perfectly and call for nothing that follows. Reading aloud is the equivalent of the render step here. It turns a text, an object you believe you know because you wrote it, into a signal you receive instead of emit. This article was reread that way.

The judge can be bought

A loop optimises what it measures. Put a judge in the stopping condition and the loop starts optimising the judge. Charles Goodhart observed this in 1975 on monetary aggregates: a statistical regularity collapses as soon as it is used as a policy target⁴. The anthropologist Marilyn Strathern gave it in 1997 the phrasing that became proverbial, from fieldwork on the assessment of British universities: when a measure becomes a target, it ceases to be a good measure⁵.

Inside a Ralph loop, the corruption of the judge takes a particular shape, and that shape has a number. Take a forecasting model improved by iteration. On each turn the agent proposes a change. You measure it on a validation set, data held back that never trains anything and only arbitrates. That measurement is noisy, as any measurement on a finite sample is: call \(\sigma\) its standard error. The ideas themselves have real effects, some good and some bad: suppose those effects spread around zero with a width \(\tau\). You try \(K\) ideas and keep the one that measures best.

The simulation above takes \(\sigma = 1\) and \(\tau = 0.3\), that is, measurement noise three and a half times larger than the gap between a good idea and a bad one, which is unremarkable on a few hundred dates. After eight hundred ideas, the change you keep shows a gain of 3.32 standard errors on validation. Its true gain is 0.27. The distance between the two curves is exactly what a fresh test set would take back, and what production always takes back.

This is no accident of the simulation, and the law behind it fits on one line. Given a measurement \(m\), the best estimate of the true effect \(\theta\) underneath it is not \(m\) but a version shrunk toward zero, shrunk further the noisier the measurement is relative to the spread of real effects:

This shrinkage is the phenomenon Bradley Efron and Carl Morris popularised as Stein's paradox⁶: a sample mean, taken on its own, systematically overstates what it measures once it was picked for being the largest. With the values in the simulation the factor is \(1 + \sigma^2/\tau^2 = 12.1\), and that is precisely the ratio observed between the two curves, at every value of \(K\), from the first attempt to the eight hundredth. How many ideas you tried does not change the factor.

It changes something else: the absolute gap. The maximum of \(K\) noisy measurements grows roughly like \(\sigma\sqrt{2\ln K}\), slowly but without ever stopping. A loop left running will therefore end up announcing an arbitrarily large gain against its own judge, having improved nothing. The same mechanism explains why so many published results fail to replicate, which John Ioannidis made quantitative in 2005 by showing that the probability a reported finding is false rises with the number of hypotheses explored before it⁷. Petroleum economists had named the thing earlier still, in 1971, noting that the winner of an auction on an oil tract is the bidder who overestimated it most: the winner's curse⁸.

From which a usable rule. The honest gain is the gain you read divided by \(1 + \sigma^2/\tau^2\), and both quantities can be estimated: the observed spread of your measurements is \(\tau^2 + \sigma^2\), and \(\sigma\) follows from the size of the validation set. More importantly, the rule gives the loop's budget a status it is never granted. Capping the number of turns is not a saving on compute. It is the condition under which the final number means anything. A Ralph loop with no budget eventually reports what you were hoping to hear.

One limit, stated plainly: this calculation is a simulation, not the measurement of a real loop. It assumes independent ideas whose effects spread around zero. That assumption is pessimistic for a competent engineer, whose proposals are not drawn at random, and optimistic for an agent producing twenty variants of a single idea, since those variants are correlated and the maximum then climbs more slowly than the formula says. It models the mechanism; it does not pass verdict on any particular loop.

The grid or the loop

The reflex for choosing a model is still the grid: enumerate the combinations of settings, evaluate them all, keep the best. James Bergstra and Yoshua Bengio showed in 2012 that at equal budget, drawing combinations at random beats a regular grid⁹, because few settings actually matter and a grid spends its points on the ones that do not.

Grid and random draw share one property: the list of candidates is fixed before a single result has been seen. A Ralph loop does the opposite, since its next proposal depends on what the last measurement said. That is adaptive analysis, and statisticians distrust it for good reason: a validation set withstands a great many questions posed in advance, and degrades far faster under questions that depend on the answers already obtained. Cynthia Dwork and her coauthors built reuse mechanisms for exactly this case, restoring a quantified guarantee to the holdout¹⁰. So one expects the loop to be the worse of the two. The simulation says otherwise.

At an equal budget of eight hundred evaluations, the grid achieves a true gain of 0.27 standard errors and the loop 1.64, six times more. The reason fits in a sentence: a grid point is a complete configuration drawn from scratch, whereas an accepted step of the loop restarts from the previous winner. Improvements accumulate in the loop and do not accumulate in the grid.

That leaves the overstatement. The grid announces 3.32 when it is worth 0.27; the loop announces 4.72 when it is worth 1.64. The two gaps are 3.05 and 3.09, which is to say the same one, and the equality holds across the whole range, not only at that value of \(K\). The overstatement therefore does not depend on how you search but on how many times you consulted the judge. It is a property of the judge, not of the search, and that simplifies the comparison: correct both numbers by the same amount, then compare them.

From which comes a budget rule you can use as it stands. Since the gap grows like \(\sigma\sqrt{2\ln K}\), requiring it to stay under \(\delta\) standard errors amounts to bounding the number of attempts:

One standard error of tolerance allows barely two attempts; two standard errors allow seven; three allow ninety. Past that, quoting the validation figure without correcting it means nothing. The formula at least gives the word budget some content instead of a vague appeal to moderation.

What really argues for the loop is not that factor of six. It is that a grid presupposes an enumerable space, and the changes that matter are not enumerable: adding a derived variable, changing the loss, rewriting the cleaning step, fixing a leak. You cannot put those in a grid, for want of axes to write. The loop searches where there is no grid to draw, and that is its irreplaceable advantage. The rest has to be paid for.

The judge is your cross-validation

Everything above rests on \(\sigma\), the judge's noise, treated so far as given. In a real project it is a quantity you fix without noticing, at the moment you choose how to split the data.

Cross-validation divides the set into \(k\) parts, trains on \(k-1\) of them, measures on the last, and repeats \(k\) times. The usual report is the mean of the \(k\) scores and an error bar:

That bar is not the uncertainty you want, and the reason is plain: dividing by \(\sqrt{k}\) assumes the \(s_j\) are independent, while any two folds share almost all of their training data. Gaël Varoquaux documented the scale of the problem on small samples, where published bars are routinely half as wide as they should be¹¹.

A measurement beats a warning. On four hundred images of handwritten digits and a logistic regression, three quantities people routinely conflate: the quoted bar is 0.0105; the variability due to the fold draw alone, with the data held fixed, is 0.0067; and the variability from one sample of four hundred rows to another is 0.0120. The last one is what limits how far a selection transfers, and it is 1.8 times the fold-to-fold spread. With stratified folds, which force every part to match the class proportions, the ratio rises to 1.95 and the quoted bar becomes 1.37 times too small: stratification tightens the fold-to-fold spread by construction, and so flatters the loop.

The consequence lands on the correction formula. Underestimating \(\sigma\) by a factor of two means underestimating the term \(\sigma^{2}/\tau^{2}\) by a factor of four, hence believing in four times more progress than there is. The splitting choice is not housekeeping: it sets the exchange rate between what you read and what you get.

That is exactly what the skill suite published by probabl around scikit-learn, skrub and skore codifies¹²: a group-wise split when several rows come from the same subject, a chronological split with an embargo when time ordering matters, and an explicit warning against stratification and against leaving out one row at a time, two habits that produce over-confident error bars.

The suite is worth reading above all as a Ralph loop written out in full. It chains data exploration, the declaration of the processing graph, a smoke test, evaluation, an audit of the resulting report, then an outer loop that sources the next experiment either from that report's diagnostics or from the user. All four conditions are met: state lives in a journal and in a design note per experiment, approved before the code exists; the judge is the report, distinct from whoever wrote the graph; every proposal arrives with the documentation reference that motivates it, which is the written reason; and several steps stop outright rather than carry on with a missing dependency or a tool name guessed from memory.

One detail there deserves copying everywhere. The smoke test is not a judgement, it is an exact assertion: the number of predictions must equal the number of rows to predict. It catches one precise fault, a graph that computes its features before splitting and silently drops its first rows for want of history to give them. That is a judgement turned back into a cheap test. Every Ralph loop gains by manufacturing as many of those as it can: what a machine can settle has no business sitting in the expensive condition.

What it does on real data

A simulation is only a simulation. Here is the same experiment run on real data, and it is harsher than the model.

Eight hundred images of handwritten digits, cut in two. Four hundred form the pool on which the judge, a five-fold cross-validation, returns its verdicts; the other four hundred make a frozen test the loop never sees. The model is a logistic regression. On each turn the loop proposes a simple, plausible change, switching three input pixels on or off, then accepts if the validation score rises and rejects otherwise. Two hundred proposals examined, ten accepted.

Validation goes from 0.9350 to 0.9650, three points gained over ten steps, each justified by a measured rise. The frozen test goes from 0.9275 to 0.9225: half a point lost. The whole of the announced progress, three and a half points counting the loss, was noise the loop had chased.

The point is not that the loop failed here. It is that it failed exactly as it would have succeeded. Compare this figure's validation curve with the one from the earlier simulation, where the proposals carried a real effect and the loop genuinely gained: they rise the same way, in steps, with nothing to mark the difference. From inside the loop the two situations are indistinguishable. Only a frozen set separates them, and it is opened once.

Forecasting without cheating

This section describes a project just starting, offered to two third-year students of the specialised master's in artificial intelligence at CentraleSupélec by Marianne Clausel, of the Université de Lorraine, Georges Oppenheim and myself. It concerns the forecasting of financial series, and its evaluation protocol is called LookForward.

The rule is short: at each origin \(t\) the model uses only the information available at that date, forecasts the next \(H\) steps, then the origin advances by one. A forecast evaluated on data it has already seen is not a forecast, it is an interpolation. The difficulty does not sit in the model, it sits in the calendar: a macroeconomic figure released on the fifteenth cannot feed a forecast made on the tenth, even though both dates fall in the same month, and an exchange closed for a holiday shifts everything after it.

Finance knows this mechanism under another name and has quantified it: David Bailey, Jonathan Borwein, Marcos López de Prado and Qiji Zhu showed that with enough trials you obtain a flattering simulated history for any strategy at all, including one with no predictive power whatsoever¹⁶. Hence the shape the protocol takes here. The project sheet specifies a budgeted Ralph loop: it proposes one change, tests it on rolling validation, then accepts or rejects it with a recorded reason. All three words carry weight. The budget bounds the \(K\) of the previous section. The recorded reason makes that \(K\) auditable afterwards, when it is usually lost. And rolling validation stays separate from a final chronological test, frozen along with its update policy and opened once, which produces the number that gets published. Read against the formula, that separation is what keeps the factor of twelve from going unnoticed.

The project compares classical statistical methods, machine-learning methods and pretrained foundation models for time series, following the open TFB benchmarking protocol¹³ on identical dates, with Chronos and TimesFM among the candidates^14, 15, and without assuming that a newer model beats plain persistence. It also plans an outcome few forecasting systems allow themselves: abstention. Conformal risk control¹⁷ supplies intervals whose error rate is guaranteed, under an exchangeability assumption that financial series, being dependent and prone to regime change, put under strain. The sheet says so itself and claims no universal per-forecast guarantee. It is the same restraint another tool on this blog, elbow-helper, applies to the elbows of curves: better to say nothing than to assert without evidence.

One item in the deliverables deserves singling out, because it is the moral of everything above. The sheet asks for an audit of figure rendering and readability using the Eyeball approach, and adds: separately from the numerical validation of the forecasts. Two different questions, two different judges. Conflating them means asking a score whether a chart is readable, or an eye whether an interval covers.

The four loops

One gesture across four materials. The fourth column is the same sentence every time, spoken in four languages.

What makes a loop hold

Four conditions emerge, and all four can be checked before anything is launched.

State lives outside the model, in files and version history, never in a conversation. The judge is independent of the actor, meaning the eye that assesses is not the hand that drew, and the validation set is not the training set. A budget bounds the number of turns, and every turn leaves a written reason, without which the \(K\) in the formula is lost and the final gain unverifiable. Abstention is a legal exit, on equal footing with success: a loop forbidden from concluding that it found nothing will always find something.

It remains to say where the likeness to the while loop stops holding, since every likeness eventually gives way. A classical loop that fails to terminate announces itself at once: the machine heats up and nothing comes out. A Ralph loop that fails to terminate produces plausible work on every turn, a slightly different figure, a slightly reworked paragraph, a slightly better score. It does not look like a breakdown. It looks like progress, and that is exactly what makes it dangerous.

The recipe fits in a sentence: render what you just made, look at it properly, fix the source and not the picture, count your turns. The first three moves cost seconds. The fourth is the one everyone forgets, and it is the only one the value of the other three depends on.

Bibliography

The references below are the ones this article actually draws on, in the order it calls them. Each note says what the source establishes and where it does its work here.

1. Huntley, G. (2025). Ralph Wiggum as a "software engineer". ghuntley.com/ralph The page that names the technique and gives its shell line. The author also states the caveats quoted here: greenfield only, a senior engineer required, output whose doneness is not guaranteed. The cost figure cited is his.
2. Harchaoui, W. The Ralph Eyeball Loop, Sprezzature. sprezzature.ai/ralph-eyeball-loop.html Applies the loop to visual artifacts and separates three complementary passes: make it from code, audit the source, look at the render. It is the source of the four steps, the eight critique headings, and the argument for grayscale as the severest test.
3. Harchaoui, W. WRITING.md, a writing charter for English prose. gist.github.com Its section 10 carries the loop over to prose: rereading by pairs of consecutive paragraphs, overlapping sliding windows, fixing the text rather than the impression. It is the charter this article applies to itself.
4. Goodhart, C. A. E. (1975). "Problems of Monetary Management: The U.K. Experience," Papers in Monetary Economics, Reserve Bank of Australia. The original observation, made on British monetary aggregates: a statistical regularity stops holding once it is used as a policy target. It grounds the section on the corruption of the judge.
5. Strathern, M. (1997). "'Improving ratings': audit in the British University system," European Review, 5(3), 305-321. Supplies the short formulation that became proverbial, out of ethnographic fieldwork on university assessment. Cited for the exact wording rather than through a secondhand quotation.
6. Efron, B. and Morris, C. (1977). "Stein's Paradox in Statistics," Scientific American, 236(5), 119-127. An accessible account of shrinkage toward the mean: a value picked because it was the largest overstates what it measures, and the correction depends on the ratio of noise to real spread. That is the boxed formula in this article.
7. Ioannidis, J. P. A. (2005). "Why Most Published Research Findings Are False," PLoS Medicine, 2(8), e124. 10.1371/journal.pmed.0020124 Makes quantitative the link between the number of hypotheses explored and the probability that a reported result is false. Same mechanism as the one simulated here, applied to the scientific literature.
8. Capen, E. C., Clapp, R. V. and Campbell, W. M. (1971). "Competitive Bidding in High-Risk Situations," Journal of Petroleum Technology, 23(6), 641-653. Origin of the phrase "winner's curse," on auctions for oil tracts: the winner is the bidder whose estimate was highest, hence most wrong. The same structure as selection on a validation set.
9. Bergstra, J. and Bengio, Y. (2012). "Random Search for Hyper-Parameter Optimization," Journal of Machine Learning Research, 13, 281-305. jmlr.org/papers/v13/bergstra12a Shows that at equal budget, drawing settings at random beats a regular grid, because performance depends strongly on only a few settings and a grid repeats its trials on the others. It is the comparison baseline for the model-selection section.
10. Dwork, C., Feldman, V., Hardt, M., Pitassi, T., Reingold, O. and Roth, A. (2015). "The reusable holdout: Preserving validity in adaptive data analysis," Science, 349(6248), 636-638. 10.1126/science.aaa9375 Establishes the difference between questions posed in advance and questions that depend on answers already obtained, and builds a mechanism that restores quantified validity to a holdout under adaptive querying. It is the theoretical frame for the grid-versus-loop distinction.
11. Varoquaux, G. (2018). "Cross-validation failure: Small sample sizes lead to large error bars," NeuroImage, 180, 68-77. 10.1016/j.neuroimage.2017.06.061 Measures the real uncertainty of a cross-validated estimate and shows it far exceeds what the fold-to-fold spread suggests, enough to cast doubt on published comparisons. It justifies the decomposition measured in this article.
12. probabl (2026). skills, agent skills for scikit-learn, skrub and skore, BSD-3-Clause. github.com/probabl-ai/skills A suite chaining exploration, construction, smoke test, evaluation and audit, with an outer loop that sources the next experiment. It serves here as an example of a Ralph loop whose judge and stop conditions are written down, notably its decision tree for choosing a splitter.
13. Qiu, X. et al. (2024). TFB: Towards Comprehensive and Fair Benchmarking of Time Series Forecasting Methods. arxiv.org/abs/2403.20150 An open benchmarking protocol for time-series forecasting, built to avoid comparisons skewed by uneven tuning. It is the evaluation frame the project described here adopts.
14. Ansari, A. F. et al. (2024). Chronos: Learning the Language of Time Series. arxiv.org/abs/2403.07815 A foundation model that treats a numeric series as a sequence of tokens, the way a language model treats text. A candidate in the comparison, with no presumption of superiority.
15. Das, A. et al. (2024). A decoder-only foundation model for time-series forecasting. arxiv.org/abs/2310.10688 The other family of pretrained models in the comparison, trained on a large corpus of series to forecast without refitting. Its size makes it the project's CPU-feasibility test.
16. Bailey, D. H., Borwein, J. M., López de Prado, M. and Zhu, Q. J. (2014). "Pseudo-Mathematics and Financial Charlatanism: The Effects of Backtest Overfitting on Out-of-Sample Performance," Notices of the American Mathematical Society, 61(5), 458-471. Quantifies backtest overfitting in finance: with enough trials you obtain a flattering simulated history for any strategy, including one with no predictive power at all. It is the financial version of the mechanism simulated above, and the reason for a frozen test.
17. Angelopoulos, A. N., Bates, S., Fisch, A., Lei, L. and Schuster, T. (2022, revised 2025). Conformal Risk Control. arxiv.org/abs/2208.02814 Extends conformal prediction to risk notions broader than coverage rate, with a guarantee that holds on average under exchangeability. That is the assumption the dependence of financial series puts under strain, as the project sheet acknowledges.

To go further, two readings wider than this article's subject: my favourite AI books for the annotated survey, and the sprezzature gallery to see what a body of figures looks like when every one of them has been through the loop.