While Not Good Yet

"While the counter is below ten, go round again." It is one of the oldest sentences in computing. It works for a reason nobody notices any more: the machine knows how to answer the question it is being asked. Comparing a counter to ten costs it one clock cycle, a single one of the operations a processor runs through billions of times per second. And the answer is exact. That is where the loop gets all its power. Its stopping condition costs almost nothing.

The questions that really decide the quality of a piece of work are not that easy. Is this chart readable? Does this paragraph follow from the one before? Does this forecasting model deserve to go to production? None of them settles in a clock cycle. So we pulled them out of our loops. We pushed them to the end, into a single review, done by hand, by someone who has already worked a lot.

The Ralph loop puts them back. It keeps the skeleton of the while loop, but it puts a judgment where the test used to be. That judge can be a human eye. It can be a vision model, an artificial-intelligence program that describes what it sees in an image. It can be a dataset held out on the side. The change looks small. It changes everything you can automate. It also opens a precise trap, one you can put a number on.

Two testing grounds stretch the loop: figures and prose. The second holds a surprise. The instrument used there was invented by Mark Twain, to mock a translation of his own story. Computer vision borrowed it from him a century later. It comes home here. As for the trap, it has a mechanism of its own: consult the same judge often enough and the loop learns the judge's quirks instead of the subject. Statisticians call that overfitting. We are going to put a number on it, watch it happen on real data, and draw two rules of conduct from it. An autonomous loop is not worth the talent of whoever acts. It is worth the quality of the judge it is given, plus the ban on corrupting that judge.

The loop in one line

The name comes from the Australian engineer Geoffrey Huntley. He described in July 2025 what he calls the Ralph technique¹. The first name is Ralph Wiggum's, the Simpsons schoolboy whose answers miss the point with admirable constancy. The technique fits on a single line, to type in a terminal:

while:; do cat PROMPT.md | claude-code; done

Word for word: while true, feed the contents of the file PROMPT.md to the agent, then start over. So you write down in a file what you want, then feed it in a loop to a coding agent. A coding agent is a language model, the program behind conversational assistants, given the right to read and write files and to run commands. You do not stop it. The surprise is that this produces usable software, rather than a heap of incoherent files.

Why does it work? A language model works inside a context window, that is, the amount of text it keeps in view during an exchange. That window is short. It also degrades as it fills: late in a long conversation, the model loses track of its own decisions. The project's files, on the other hand, do not degrade. On every turn of Huntley's loop, the agent starts from scratch, rereads the real state of the files, does one thing, writes it down. Progress accumulates on disk, never in the model's memory. Forgetting is what makes the technique work.

Huntley does not sell his technique as a solution. He writes that it is "deterministically bad in a non-deterministic world." He adds that it only suits a brand-new project, with an experienced engineer at the helm. Its output comes out, he says, "undercooked, cooked or cooked with unspecified latent behaviours." He reports a contract billed 50,000 dollars and delivered for 297 dollars of compute: that figure is his, and nothing in this article verifies it.

One detail of his command line deserves a stop, because it is this article's whole subject. while: means "while true": the condition is empty. The original loop has no test at all. It runs until a human stops it. Everything interesting begins when you put something back into that condition. Huntley himself extended the idea in January 2026, arguing it holds for every task², without saying how. That is what the rest of this article tries to do.

The cycle: act, render, judge, decide

A four-step cycle is enough to describe every variant you will meet. Act: produce one change, a single one, so you know later what to credit for the result. Render: turn that change into an object you can put in front of you. An image, a printed text, a table of measurements. Judge: look at that object. Decide: keep or discard, writing down why.

The step we skip most often is the second, because it seems the most useless. Without a render, we judge an intention and not a result: the loop spins without learning anything. The fourth is the one we regret. Without a written reason, we lose the count. Three weeks later nobody knows how many ideas were tried before the one that survived. That count of attempts is not paperwork. It is the number on which the validity of everything else depends.

The idea is not new, which is rather reassuring. It simply goes by a different name in every trade. Software engineering has known it for thirty years as the oracle problem: how do you tell correct behavior from faulty behavior, when no specification does it for you? A 2015 survey concludes that the last source of truth remains the human³. Machine learning rediscovered it through failure: a language model left alone corrects itself poorly, and sometimes degrades in the attempt⁴. A July 2026 survey of 1,250 papers sums it all up in one sentence: every improvement loop is a bet that some signal can replace human judgment⁵. The sturdier the signal, the better the loop. So the question moves from the agent to the judge it is given.

First testing ground: figures

You would expect, in 2026, any model to produce a professional figure unaided. In reality, figures remain among the hardest objects to delegate: an image that came out of a machine is recognizable at first glance. It is that gap between the promise and the result that makes figures the loop's first testing ground.

A sentence is checked by reading it again. A chart is not. The code that draws a figure can be entirely correct and the figure wrong all the same. A label slides out of frame. A legend sits on top of the data. Two points land exactly on each other. A red-and-green palette collapses into one muddy hue, for anyone who cannot tell those two colors apart. None of that shows up when you read the source. It shows up when you look at the rendered result.

That is the discipline the Ralph Eyeball Loop⁶ formalizes, at sprezzature.ai. Render, look, fix the source, repeat. Until there is nothing left to spot. The tool turns the source into an image with an engine that never calls on randomness: the same source therefore always gives the same image. It then keeps, for each figure, a critique log under eight required headings. Composition, contrast, visual hierarchy, spacing. Accessibility, color, label legibility at the size it is rendered, and an overall verdict. That log grows with every pass instead of being overwritten: it keeps the history of what was seen and fixed. The sprezzature gallery holds one hundred and twenty-seven kinds of charts, every one of them put through this loop.

Who looks? By default, the agent that just wrote the code looks at its own output. An option hands the first pass to a vision model installed locally, which a human then rereads. The difference matters, because a machine judge can be measured. A benchmark published in September 2026 took multimodal models, the ones that read images as well as text, and put them to work judging charts. It finds systematic weaknesses in them, the first being a strong tendency to accept everything⁷. One detail matters for what follows. Some models were rewarded during training when their answer pleased. Those are the most lenient of all. The judge therefore bends under the very pressure it is supposed to referee. A human eye has to stay in the loop.

A concrete case says it better than a lot of theory. This one comes from this very page. The articles on this blog descend from a layout tool's export. The stylesheet fixes the page's appearance. It declared a spacing between paragraphs. Another rule reset it to zero. The result: across the eight published articles, blocks of text were touching. Everything was valid, no test failed, no automated audit complained. Someone had to photograph the rendered page and look at it. The same pass surfaced two more defects of the same kind: bibliography numbers sitting on the first letter of their note and stray blank lines between list items. Three defects visible in one second of looking, invisible in any amount of reading the code.

The loop earns its keep where the judge sees what the actor cannot. A person who tells colors apart normally may take all the care in the world: they will never see their own figure the way a color-blind reader sees it. The information is not in their eye. So a simulator redraws the image as those readers perceive it, then in black and white. Black and white is in fact the harshest test: a figure that reads there reads under every other deficiency. If the point survives gray, it survives everyone. And that check takes a second, which makes it a stopping condition almost as cheap as comparing a counter to ten.

Second testing ground: prose

None of this is specific to figures. Change the judge in the stopping condition: the same skeleton applies to text. The writing charter this article answers to makes it a rule⁸. You do not reread a text paragraph by paragraph, but by pairs of paragraphs that follow one another. The reason is simple: what catches a reader is not inside a paragraph, it is at the seam between two. So you read paragraph n then paragraph n+1 aloud, in one go. You look at the crossing from one to the other. Then you fix the text itself: the order of sentences, the last line of the first, the first line of the second. You do not just note in the margin that "something sticks". Then you slide one notch, to the next pair.

The windows overlap, so each paragraph is tested twice. Once as an arrival, once as a departure. Reading aloud plays the role of the render here. You think you know your own text, since you wrote it; the voice turns it into a signal you receive instead of emit. This article went through that.

Here too, the danger has been measured rather than feared. Jane Pan, He He, Samuel Bowman and Shi Feng showed it in 2024. In a loop where the same model writes and grades, the grade climbs while quality as judged by real readers stalls or falls. The effect is worst when writer and grader share their context⁹. That is the sharpest argument for a reread by someone who did not write the text.

Reading aloud has a blind spot, though. It tests the seam between two paragraphs, which only shows at the crossing from one to the other. It does not test the inside of either. An ambiguous sentence you wrote yourself reads smoothly, because you already know what it means. The ear cannot catch what memory fills in. So a second instrument is needed, working on the same unit but looking elsewhere.

That instrument is called ABABA and its name describes the operation. You write a passage in French, translate it into English, translate it back into French, then start again. A for one language, B for the other. ABABA therefore counts five states, four translations, two full round trips. The word is a palindrome: it reads the same in both directions. That is exactly what we are trying to check on the text.

Write \(T_{AB}\) for the translation into English and \(T_{BA}\) for the return trip. One full round is their composition \(R = T_{BA} \circ T_{AB}\). The sequence of successive versions is then:

A well-written passage therefore comes back from the trip intact, because its meaning is carried by its words and not by the reader's goodwill. Mathematicians say it is a fixed point of \(R\). Machine learning calls that property cycle consistency. It took a hundred-year detour through paintings and horses before coming back to text.

The journey begins with a writer's revenge. In 1903, Mark Twain published his jumping-frog story in three versions. His own, first. Then the French translation Thérèse Bentzon had made of it. Then his own word-for-word retranslation of that French back into English, French syntax and all¹⁰. The result is grotesque. That is the point: Twain shows what the trip did to his text. The device turned serious in 1970, when Richard Brislin made it the standard quality control of cross-cultural research¹¹. You translate a questionnaire, you translate it back. The gap then names the badly worded question, before it skews a clinical study.

Next come the horses. In 2017, Jun-Yan Zhu, Taesung Park, Phillip Isola and Alexei Efros set out to turn photographs of horses into zebras¹². The problem: no matching pairs of photographs exist, so there is no answer key. Their solution was to require that a round trip give back the starting image. Efros says where the idea came from. The answer closes the circle¹³: from Mark Twain and from back-translation. He gives the version anyone understands. To have a text rendered into a language you do not speak, you pay one translator for the outward leg and a second for the return, then you check that the thing still stands up. The technique was born in text, then left for images. We are bringing it home.

Efros also supplies the image that explains why a round trip teaches you anything. Think of a setup that compresses a piece of data then rebuilds it. Here, the narrow passage in the middle is not a compression: it is another language. The text is forced through a foreign language, then back. And that passage throws a great deal away. It throws away the turns of phrase that are ours without meaning anything. It throws away the word order we had thought elegant. Above all, it cannot render an ambiguity: the translator has to pick one reading and stick to it.

A sentence that comes back deformed therefore marks a place where the text does not constrain its own reading enough. It may be an ambiguity. It may also be an idea left too vague, a turn of phrase the other language will not carry, a usage that differs between languages, or simply a translator's taste. The loop does not tell these apart: it points at the spot. Going to look is our job.

It does see what we no longer can. We already knew what we meant, whereas the translator does not: it is the first naive reader we have on hand. An acronym never explained. A term defined nowhere. A pronoun whose referent we have forgotten three sentences back. None of the three survives the trip.

Stability alone proves nothing, and Efros says so himself in the same talk. If all you ask is a return to the starting point, the network can cheat by not moving at all: changing nothing cycles perfectly. So a second requirement had to be added: a transformed horse really has to look like a zebra. The measured version of that cheat arrived the same year. Casey Chu, Andrey Zhmoginov and Mark Sandler showed that CycleGAN learns to hide the source image inside invisible noise added to its own output¹⁴. The cycle closes flawlessly without anything having been translated. They called it steganography, the art of hiding a message inside an innocuous-looking carrier.

Hence a two-part criterion, neither part optional. Stability is measured against the previous round: is the text still moving? Fidelity is measured against the original, always, and never against the previous round. In practice you list what the starting text claims, list what the arriving text claims, then compare the two lists. There is a reason for that asymmetry. A drift in meaning advances through small steps, invisible one at a time but considerable once added up. It also hides itself: by the second round the text has become coherent with its own error.

This article's central trap returns here in a sharp form. By repeating the round trip, the text is pulled towards one particular state, whatever its starting point. That state is the translator's average prose. Iterating until the text stops moving therefore means arriving there: flattened vocabulary, dead rhythm, bolted-on transitions. The loop clarifies and flattens in one motion, without being able to separate the two. A neighboring phenomenon has been measured elsewhere. Ilia Shumailov and his coauthors showed in 2024 that a model retrained on its own outputs first loses whatever was rare in its answers¹⁵. The comparison has a limit worth stating: their loop retrains the model, ours only queries it. What grows poorer here is not in the model, it is in the text.

The remedy is simple: never keep the loop's output. Keep whatever moves from one round to the next, the way you would keep a map. The passages that refuse to settle are the badly written ones: those are the ones you rewrite by hand. It is the instability that is the signal, not the final text. If a round turns "X" into "Y", the useful information is not Y. It is that X could be rewritten, so a reader could have understood it as Y. That was all we wanted to know. What comes next is written in our own hand, which protects the style.

Alternating the translators costs little, and Twain and Efros already did it by hiring two different people. With a single model, the text settles on that model's style. With two models in alternation, a text that settles has to satisfy both sets of preferences. That pulls it away from either one's habits. That is an intuition, not a proved property. The language you pass through deserves the same care. French and English share too much structure: a calque slips across unresisted, and a badly built French sentence stays badly built in English without anything sounding wrong. For diagnosis, a genuinely distant language such as Japanese or Finnish forces a real re-encoding.

A serious objection deserves to be raised rather than buried. Harold Somers showed in 2005 that the round trip is a poor way to grade a machine translator¹⁶. Errors on the way out and errors on the way back cancel each other, so a successful round proves nothing about the quality of either translation. The objection is sound, but it does not apply here, because what we are looking for has moved. Somers varies the translator and holds the text fixed, since he wants to grade the translator. We do the opposite: the translator is fixed and assumed good, while the text varies.

Two recent measurements confirm that the method holds, when approached this way. Jihyung Moon, Hyunchang Cho and Eunjeong Park showed in 2020 that the round trip predicts quality well, provided you compare meanings rather than words¹⁷. In April 2026, Ronald Skorobogat, Ameya Prabhu and Matthias Bethge measured a correlation of 0.94 between a round-trip score and the preferences of real users. Two established multilingual benchmarks, meanwhile, fall to zero or turn negative¹⁸. The error cancellation Somers describes is still a good reason never to conclude from a single round with a single model.

Two instruments, then, on the same unit of text. The ear tests the seam, translation tests the inside. You read aloud first, because it is free and settles half the problems. You launch the round trip only on the pairs that resist, and never more than two rounds on a passage that yields. Past five rounds, you are measuring the translator's style and not the text.

The trap: the judge ends up bought

A loop optimizes what it measures. Put a judge into the stopping condition. The loop then starts optimizing that judge. Charles Goodhart observed it in 1975 on British monetary policy: a statistical regularity collapses as soon as it is used as a target¹⁹. The anthropologist Marilyn Strathern gave it in 1997 the wording that stuck, while studying university assessment: when a measure becomes a target, it ceases to be a good measure²⁰. Then came the modern version, measured on language models. Leo Gao, John Schulman and Jacob Hilton optimized harder and harder against an artificial judge. The awarded score climbs without stopping. True quality climbs, plateaus, then falls²¹. The judge does not break at once: it bends as you push, which is far harder to notice.

In a Ralph loop, that corruption can be priced. Take a forecasting model being improved step by step. On each turn, the agent proposes a change. You measure it on a validation set: data held out on the side, never used for training, only for arbitration. That measurement is noisy, like every measurement made on a limited amount of data. Redone on other data, it would move. Call \(\sigma\) the typical size of those fluctuations. The ideas, for their part, have real effects, sometimes good, sometimes bad, of typical size \(\tau\). You try \(K\) ideas, then keep the one that measures best.

The simulation above takes measurement noise three and a half times larger than the gap between a good idea and a bad one, which is nothing extravagant in practice. After eight hundred ideas tried, the winning change shows a gain of 3.32 on validation. Its true gain is 0.27: twelve times less. The gap between the two curves is exactly what going to production always comes to claw back.

Nothing accidental there. Think of a competitive exam. The student who takes the top mark generally has both ability and luck: picking the best paper also picks the luck. The best estimate of their true level is therefore a little below their mark. A formula prices that discount. Given a measurement \(m\), the best estimate of the true effect \(\theta\) is not \(m\): it is a version of \(m\) pulled toward zero, and pulled harder when the measurement is noisy:

This is the Bayesian shrinkage formula, from the family of estimators made famous by James and Stein in the 1950s²². Bradley Efron tied it in 2011 to selection bias²³: a value picked because it was the largest always overstates what it measures. With the simulation's values, the factor is 12.1: whatever validation announces, divide it by twelve. The number of ideas tried changes nothing about that factor. It changes the size of the mirage, which grows like \(\sqrt{2\ln K}\), a function that climbs very slowly but never stops. A loop left running therefore ends up announcing as large a gain as you like on its judge, without having improved anything. The same mechanism explains why so many published results fail to replicate, as John Ioannidis showed in 2005²⁴. Petroleum economists had already named it in 1971: the winner of an auction is the bidder who most overestimated the tract, hence the winner's curse²⁵.

One limit to state right away: this calculation is a simulation, which assumes independent ideas with no average talent. That is pessimistic for a competent engineer, whose proposals are not drawn at random. It is optimistic for an agent producing twenty variants of the same idea. It is a model of the mechanism, not a verdict on any particular loop. What survives those caveats fits in two sentences. What you read exceeds what you get. And the gap widens with the number of tries.

The same thing, on real data

A simulation is still a simulation. Here is the same experiment on real data. It is harsher than the model. Eight hundred images of handwritten digits, cut in two. Four hundred serve the judge, through cross-validation, whose workings the next section explains. The other four hundred form a frozen test the loop never sees. On each turn, the loop proposes a small change to the input variables. It accepts the change if the validation score rises, rejects it otherwise. Two hundred proposals examined, ten accepted. The full protocol is given so it can be redone rather than believed, seed included. The seed is the number that pins down the randomness: rerun the program and you get exactly the same experiment. The load_digits dataset from scikit-learn, seed 20260922, logistic regression, five-fold cross-validation.

Validation gains three points, in ten steps, each one justified by a measured rise. The frozen test loses half a point. All the announced progress was noise the loop chased. The important point is not that the loop failed here. It is that it failed exactly the way it would have succeeded. Its validation curve climbs in steps, like the curve of a loop that is genuinely winning. Seen from the inside, the two situations cannot be told apart. Only a frozen set, consulted once, separates them.

A second objection weighs more than the first. The largest empirical check points the other way. Rebecca Roelofs and her coauthors studied more than a hundred Kaggle competitions. They compared the public leaderboards, built from thousands of submissions, against the private leaderboards revealed once at the end. They found little overfitting²⁹. The explanation fits in two words: effective multiplicity. Twenty variants of the same idea do not count as twenty independent tries. The handwritten-digits experiment, on the other hand, draws nearly independent tries: it is the unfavourable case, chosen on purpose. So the right question is not whether a loop overfits, but how many genuinely different tries it has made.

The budget or what makes a number believable

Should you then give up searching in a loop? No, and a comparison shows why. The classic reflex for tuning a model is the grid: you write a list of setting combinations in advance, evaluate them all, keep the best. James Bergstra and Yoshua Bengio showed in 2012 that at equal budget, drawing the combinations at random already beats a regular grid²⁶. A Ralph loop goes further: its next proposal depends on what the previous measurement said. Statisticians call that adaptive analysis and rightly distrust it. A validation set wears out much faster when the questions depend on the answers already obtained. Cynthia Dwork and her coauthors built quantified guarantees for that case²⁷. You would therefore expect the loop to do worse than the grid. The simulation says the opposite.

At equal budget, the grid obtains a true gain of 0.27, the loop 1.64: six times more. Where does the advantage come from? A third experiment settles it, playing the role of the control group in a drug trial. The same step-by-step walk, but accepting everything without ever looking at the measurement, brings nothing. So it is the selection that does the work, not the accumulation. Conversely, the same walk judged by an oracle that could see the true effects would gain sixty times more. With its noisy judge, the loop recovers only two percent of what was available. As for the mirage, it is the same on both sides: grid and loop overstate their gain by as much. The inflation does not depend on how you search, only on how many times you consulted the judge.

From this comes a budget rule, whose address has to be given precisely, because it is not mine. In 2014, David Bailey, Jonathan Borwein, Marcos López de Prado and Qiji Zhu treated this exact problem for financial backtests²⁸. A backtest is a simulation that replays an investment strategy on the past. Their result: below a certain amount of data, a flattering track record proves nothing. With five years of history, you should not try more than forty-five different configurations. Carried over to a loop that accepts an optimism of \(\delta\), the same bound gives a cautious rule:

Cautious, because the formula overshoots. The exact calculation allows about three tries for a tolerance of 1, twenty-seven for 2, four hundred and forty-three for 3. The formula concedes only one, seven and ninety. And \(K\) must be counted in genuinely different tries, not in loop turns. One point survives every caveat, proven since 2014 in an entirely different field. Capping the number of turns is not a compute saving. It is the condition for the final number to mean anything. A loop without a budget ends up reporting what you hoped to hear.

What really tips the scale toward the loop is not that factor of six. It is that a grid assumes you can write the list of candidates in advance. But the changes that matter cannot be written as a list. Adding a derived variable. Changing the loss function. Fixing a data leak. You cannot put those in a grid, because you cannot write its axes. The loop searches where there is no grid to draw. That is its irreplaceable advantage. The rest has a price.

The everyday judge: cross-validation

Everything above rests on \(\sigma\), the judge's noise. In a real project, that quantity gets set without anyone noticing, at the moment the data split is chosen. The standard tool is cross-validation. You split the dataset into \(k\) parts, called the folds. You train on all the folds but one, then measure on the one left over. You start again \(k\) times, changing which fold is held out. You then quote the mean of the \(k\) scores, with an error bar meant to say what that mean is worth.

That bar is almost always too optimistic, for a simple reason: the calculation acts as if the \(k\) measurements were independent, when two folds share nearly all their training data. Gaël Varoquaux documented the size of the problem: on small samples, published bars are routinely twice too narrow³⁰. On the handwritten-digits example, the variability that matters, from one sample to the next, is nearly double the one quoted between folds. And underestimating \(\sigma\) by a factor of two means believing in four times more progress than there is. The split is therefore not a logistical detail: it sets the exchange rate between what you read and what you get.

That is what the tool suite published by probabl around scikit-learn, skrub and skore codifies³¹. A split by groups when several rows come from the same subject. A chronological split with a buffer zone when time order matters. And an explicit warning against the habits that produce overconfident bars. The suite reads like a Ralph loop written out in full. State lives in a logbook. The judge is a report distinct from whoever wrote the pipeline. Every proposal arrives with its written reason. Several steps stop dead rather than carry on over a missing dependency.

One detail of that suite deserves copying everywhere. Its smoke test is not a judgment, it is an exact check: the number of predictions must equal the number of rows to predict. It catches one precise, frequent fault. There is a judgment turned back into a test that costs nothing. Every Ralph loop gains from manufacturing as many of these as it can: whatever the machine can settle has no business sitting in the expensive condition.

Forecasting without cheating

This section describes a project just starting, proposed to two students of the specialized master's in artificial intelligence at CentraleSupélec by Marianne Clausel, of the Université de Lorraine, Georges Oppenheim and myself. It concerns forecasting financial series, and its evaluation protocol is called LookForward. The rule is short. You stand at a date, which we call the origin. The model uses only the information available on that date, then predicts the next steps. Then the origin advances one notch. A forecast evaluated on data it has already seen is not a forecast, it is an interpolation. The difficulty is not in the model, it is in the calendar: an indicator published on the fifteenth cannot serve a forecast made on the tenth.

Finance is the very field where Bailey and his coauthors showed that with enough tries, you get a flattering track record for any strategy, even one with no predictive power at all²⁸. Hence the protocol's shape. A Ralph loop under budget tests each change in rolling validation, then accepts or rejects it while writing down why. Kept apart from all of it, a final chronological test, frozen, opened once, yields the number that will be published. The budget caps the number of tries. The written reason makes the loop checkable. And the frozen test keeps the factor of twelve from slipping by unnoticed.

The project compares three families. Classical statistical methods. Machine-learning methods. And time-series foundation models, those large models already trained on huge collections of series and reused as they are. It follows the open protocol TFB³², with Chronos and TimesFM among the candidates^33, 34. Nothing in it assumes that a recent model beats the most naive forecast there is: tomorrow will be worth what today is worth. It finally allows itself a rare exit, abstention, backed by conformal risk control³⁵. That guarantee rests on an assumption financial series strain. The project sheet says so itself. It is the same restraint that elbow-helper applies to curve elbows: better to stay silent than to assert without proof.

One detail of the deliverables sums up everything above. The sheet asks that the figures' legibility be checked with the Eyeball approach, separately from the numerical validation of the forecasts. Two different questions, two different judges. Confusing them means asking a score whether a chart is readable, or asking an eye whether a range really covers the truth.

One cycle across four testing grounds

One cycle across four testing grounds. The fourth column is the same sentence every time, spoken in four languages.

Where the idea comes from and what this article adds

A clarification, because the word is starting to mean everything. Chaining production, critique and correction is in no way specific to Ralph, and predates it. Self-Refine has the model critique itself³⁶. Reflexion keeps a memory of comments from one attempt to the next³⁷. What is specific to Huntley's technique is a combination. A fresh context on every turn, rather than a lengthening conversation. State that lives outside the model. Real execution of the code produced. One single thing done per turn. As for the loops on prose, figures and forecasting described above, they are my generalization, not an established sense of the term. A reader searching for "Ralph loop" elsewhere will find code and nothing else.

None of the numbered part is mine either. Quantitative finance proved in 2014 that choosing among \(N\) backtests imposes a minimum length of history²⁸. Theoretical computer science proved in 2015 that a held-out set only survives a limited number of adaptive questions²⁷. The alignment community measured, between 2022 and 2024, that pushing too hard against a stand-in judge destroys real quality²¹. Shrinkage toward the mean dates from the 1950s. Meanwhile, in 2025, software engineering put into production a loop with no stopping condition at all. These literatures do not cite one another. This article's only contribution is to say that they describe the same object, with the budget rule holding for agent harnesses exactly as it does for market strategies.

The guards

The cycle comes with five guards, to check before launching anything. State lives outside the model, in files and their history, never in a conversation. The judge is independent of the actor. The eye that evaluates is not the hand that drew. The validation set is not the training set. A budget caps the number of turns. Every turn leaves a written reason, without which you lose the count of tries and the final gain becomes uncheckable. And abstention is a permitted exit, on the same footing as success: a loop that is not allowed to conclude it found nothing will always find something.

Those five guards amount to a working definition, better than the word "loop" on its own. A good agent loop is not a "retry until it passes". It is a search that adapts. Its state survives outside the model. It is judged from the outside. It has an announced budget and the right to stay silent. Drop a single one of those guards: the final number is no longer checkable.

They all obey one principle, which Efros puts better than I can in the talk already cited¹³. You do not build an incorruptible judge: no such thing exists. The data is finite, so there will always be some path for cheating, and plugging one hole opens another. What you can do is arrange the problem so that cheating costs more than working. He uses the image of a well-run country: you can break the law there, you simply have no interest in doing so. In a badly run country, cheating is the rational calculation, and everyone cheats.

His example comes from his own classroom. A whole semester of work is expensive. Memorizing last year's past papers is cheap. And it often suffices to pass the exam. The student who makes that calculation is not dishonest, only rational. It is the exam that is badly built. A validation set consulted two hundred times is exactly those past papers. The budget, the written reason and the frozen test do not moralise the loop. They make cheating more expensive than working.

One last thing. Here is where the analogy with the while loop finally gives way. A classic loop that fails to stop is noticed immediately: the machine heats up and nothing comes out. A Ralph loop that fails to stop produces plausible work on every turn, a slightly different figure, a slightly reworked paragraph, a slightly better score. It does not look like a breakdown. It looks like progress. That is exactly what makes it dangerous.

Hence the closing recipe: render what you just made, actually look at it, fix the source and not the image, count your turns. The first three steps cost seconds. The fourth is the one everyone forgets and the only one the other three depend on. Because an autonomous loop is not worth the quality of its actor: it is worth the evaluation architecture it is forbidden to overfit.

Bibliography

Every title below went into writing one particular section; the note says which. None is cited second-hand.

1. Huntley, G. (2025). Ralph Wiggum as a "software engineer". ghuntley.com/ralph The page that names the technique and gives its shell line. The author also states the caveats quoted here: greenfield only, a senior engineer required, output whose doneness is not guaranteed. It carries no stopping condition at all: the operator decides. The cost figure cited is his.
2. Huntley, G. (2026). everything is a ralph loop. ghuntley.com/loop Argues the pattern holds for all tasks and sketches several orchestration variants. It is the most direct antecedent of the generalization defended here; generality is asserted rather than structured, with no common skeleton and no failure-mode analysis.
3. Barr, E. T., Harman, M., McMinn, P., Shahbaz, M. and Yoo, S. (2015). "The Oracle Problem in Software Testing: A Survey," IEEE Transactions on Software Engineering, 41(5), 507-525. 10.1109/TSE.2014.2372785 Surveys thirty years of attempts to decide automatically whether observed behavior is correct and concludes that where specifications, models and metamorphic testing stop, the last source of oracle information remains the human. This is software engineering's statement of the problem, long predating agents.
4. Huang, J., Chen, X., Mishra, S., Zheng, H. S., Yu, A. W., Song, X. and Zhou, D. (2024). Large Language Models Cannot Self-Correct Reasoning Yet, ICLR. arxiv.org/abs/2310.01798 Separates self-correction from the model's own resources alone from correction leaning on an outside signal and shows the former fails, performance sometimes degrading after correction. It is the empirical case for the judge coming from somewhere other than the actor.
5. Chen, M., Wang, L. and Qu, B. (2026). Recursive Self-Improvement in AI: From Bounded Self-Refinement to Autonomous Research Loops. arxiv.org/abs/2607.07663 A survey of 1,250 papers holding that every improvement loop is a claim that some signal can substitute for human judgment and ranking those signals from formal verifier to intrinsic self-assessment. It is the closest statement to this article's framing and it came first of the two by several months.
6. Harchaoui, W. The Ralph Eyeball Loop, Sprezzature. sprezzature.ai/ralph-eyeball-loop.html Applies the loop to visual artifacts and separates three complementary passes: make it from code, audit the source, look at the render. It is the source of the four steps, the eight critique headings and the argument for grayscale as the severest test.
7. Wu, L., Zhao, H. H., Zhang, Z., Tang, J., Wu, J. and Wang, A. J. (2026). ChartJudgeBench: Evaluating LMM Judges for Chart-to-Code Generation. arxiv.org/abs/2609.24210 Evaluates multimodal models used as chart judges and documents four systematic weaknesses, among them a marked tendency to accept and greater leniency in reinforcement-trained models. It is the measurement of the visual judge the figure loop depends on.
8. Harchaoui, W. WRITING.md, a writing charter for English prose. gist.github.com Section 10 carries the loop over to prose: rereading by pairs of consecutive paragraphs, overlapping sliding windows, fixing the text rather than the impression. It is the charter this article applies to itself.
9. Pan, J., He, H., Bowman, S. R. and Feng, S. (2024). Spontaneous Reward Hacking in Iterative Self-Refinement. arxiv.org/abs/2407.04549 Shows that in a loop where one model both writes and grades, the grade rises while quality as perceived by real readers stalls or falls, the more so when writer and grader share context. The failure mode described here was therefore already named and measured on prose.
10. Twain, M. (1903). The Jumping Frog: In English, Then in French, Then Clawed Back into a Civilized Language Once More by Patient, Unremunerated Toil, Harper & Brothers. Twain prints his story, the French translation Thérèse Bentzon made of it and his own word-for-word retranslation, side by side. The demonstration is comic and exact: the round trip makes visible whatever was fragile in the text. It is the literary ancestor of the loop described here.
11. Brislin, R. W. (1970). "Back-Translation for Cross-Cultural Research", Journal of Cross-Cultural Psychology, 1(3), 185-216. 10.1177/135910457000100301 Establishes back-translation as quality control in cross-cultural research; it has been the standard procedure ever since for translating a clinical or psychometric questionnaire, where one lost nuance skews a whole study. It settles that the return gap is an exploitable signal rather than noise.
12. Zhu, J.-Y., Park, T., Isola, P. and Efros, A. A. (2017). "Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks", ICCV, 2242-2251. arXiv:1703.10593 CycleGAN: with no matching image pairs available, the training constraint becomes the round trip itself. It is the formalization of cycle consistency used here, together with the proof that it can stand in for a missing answer key.
13. Efros, A. A. (2018). Self-supervision, Meta-supervision, Curiosity: Making Computers Study Harder, lecture at CIIRC, Czech Technical University in Prague, 25 May 2018. youtube.com/watch?v=_V-WpE8cmpc Efros explicitly credits CycleGAN's idea to Mark Twain and back-translation and describes the setup as an autoencoder whose narrow passage is another language (at 48 min). He also states the limit taken up here: without a second constraint, the network cheats by not moving. And the formulation that carries the end of this article: make cheating more expensive than doing the work.
14. Chu, C., Zhmoginov, A. and Sandler, M. (2017). CycleGAN, a Master of Steganography, "Machine Deception" workshop, NIPS. arXiv:1712.02950 Shows that CycleGAN conceals the source image in an imperceptible high-frequency noise, which lets it reconstruct exactly without having translated anything. It is the demonstration that cycle consistency alone is degenerate. Hence fidelity is measured here against the original, never against the previous round.
15. Shumailov, I., Shumaylov, Z., Zhao, Y., Papernot, N., Anderson, R. and Gal, Y. (2024). "AI models collapse when trained on recursively generated data", Nature, 631, 755-759. 10.1038/s41586-024-07566-y Measures what becomes of a model retrained on its own outputs: the tails of the distribution go first, which is to say whatever was rare. Cited here as a measured analog of the attractor, with its limit stated plainly: theirs is a retraining loop, not a querying loop.
16. Somers, H. (2005). "Round-trip Translation: What Is It Good For?", Australasian Language Technology Workshop, 127-133. aclanthology.org/U05-1019 The substantive objection: the round trip grades a machine translator very poorly, since errors on the outward and return legs cancel. Taken up here rather than set aside; it does not target the same use, but it directly motivates alternating translators.
17. Moon, J., Cho, H. and Park, E. L. (2020). "Revisiting Round-Trip Translation for Quality Estimation", EAMT. arXiv:2004.13937 Rehabilitates the round trip by changing what gets compared: meaning embeddings instead of word overlap. It is the technical answer to Somers, hence the justification for the fidelity criterion used here, which bears on claims and not on form.
18. Skorobogat, R., Prabhu, A. and Bethge, M. (2026). Round-Trip Translation Reveals What Frontier Multilingual Benchmarks Miss, 14 April 2026. arXiv:2604.12911 Measures a correlation of 0.94 between a round-trip score and user preferences, where two established multilingual benchmarks fall to −0.09 and −0.26. It is the most recent evidence that the round trip catches something real. It bears on models rather than on text, so it supports the use made here without proving it.
19. Goodhart, C. A. E. (1975). "Problems of Monetary Management: The U.K. Experience," Papers in Monetary Economics, Reserve Bank of Australia. The original observation, made on British monetary aggregates: a statistical regularity stops holding once it is used as a policy target. It grounds the section on the corruption of the judge.
20. Strathern, M. (1997). "'Improving ratings': audit in the British University system," European Review, 5(3), 305-321. Supplies the short formulation that became proverbial, out of ethnographic fieldwork on university assessment. Cited for the exact wording rather than through a secondhand quotation.
21. Gao, L., Schulman, J. and Hilton, J. (2023). "Scaling Laws for Reward Model Overoptimization," ICML, PMLR 202, 10835-10866. arxiv.org/abs/2210.10760 Has a reward model stand in for the human judge and measures what happens as you optimize against it: the score climbs, true quality plateaus then falls, along a fitted functional form. Goodhart's law made quantitative, on a subject this article does not otherwise treat.
22. Efron, B. and Morris, C. (1977). "Stein's Paradox in Statistics," Scientific American, 236(5), 119-127. An accessible account of shrinkage toward the mean: a value picked because it was the largest overstates what it measures. How much to take back depends on the ratio of noise to real spread. That is the boxed formula in this article. It dates from the 1950s.
23. Efron, B. (2011). "Tweedie's Formula and Selection Bias," Journal of the American Statistical Association, 106(496), 1602-1614. 10.1198/jasa.2011.tm11181 Ties shrinkage explicitly to selection bias: the largest few estimates overstate their parameters, with the correction written from the observed marginal density. It is the exact address of the factor used here.
24. Ioannidis, J. P. A. (2005). "Why Most Published Research Findings Are False," PLoS Medicine, 2(8), e124. 10.1371/journal.pmed.0020124 Makes quantitative the link between the number of hypotheses explored and the probability that a reported result is false. Same mechanism as the one simulated here, applied to the scientific literature.
25. Capen, E. C., Clapp, R. V. and Campbell, W. M. (1971). "Competitive Bidding in High-Risk Situations," Journal of Petroleum Technology, 23(6), 641-653. Origin of the phrase "winner's curse," on auctions for oil tracts: the winner is the bidder whose estimate was highest, hence most wrong. The same structure as selection on a validation set.
26. Bergstra, J. and Bengio, Y. (2012). "Random Search for Hyper-Parameter Optimization," Journal of Machine Learning Research, 13, 281-305. jmlr.org/papers/v13/bergstra12a Shows that at equal budget, drawing settings at random beats a regular grid, because performance depends strongly on only a few settings and a grid repeats its trials on the others. It is the comparison baseline for the model-selection section.
27. Dwork, C., Feldman, V., Hardt, M., Pitassi, T., Reingold, O. and Roth, A. (2015). "The reusable holdout: Preserving validity in adaptive data analysis," Science, 349(6248), 636-638. 10.1126/science.aaa9375 Separates questions posed in advance from questions depending on answers already obtained and builds a mechanism restoring quantified validity to a holdout under adaptive querying, at the cost of an explicit budget of disagreements. It is the theoretical-computer-science proof that the budget is a necessity, not a prudence.
28. Bailey, D. H., Borwein, J. M., López de Prado, M. and Zhu, Q. J. (2014). "Pseudo-Mathematics and Financial Charlatanism: The Effects of Backtest Overfitting on Out-of-Sample Performance," Notices of the American Mathematical Society, 61(5), 458-471. 10.1090/noti1105 Computes the expected maximum of \(N\) skill-free trials, bounds its growth by \(\sqrt{2\ln N}\) and derives a minimum-backtest-length theorem. It is the source of this article's budget rule, which only carries it across; the precedence is total, the bound itself coming from extreme-value theory.
29. Roelofs, R., Shankar, V., Recht, B., Fridovich-Keil, S., Hardt, M., Miller, J. and Schmidt, L. (2019). A Meta-Analysis of Overfitting in Machine Learning, NeurIPS. Compares, across more than a hundred Kaggle competitions, public leaderboards built from thousands of submissions against private leaderboards revealed once: the observed overfitting is small, if anything. It is the empirical counter-evidence to the worst-case theory and the reason to speak of effective multiplicity rather than of turn count.
30. Varoquaux, G. (2018). "Cross-validation failure: Small sample sizes lead to large error bars," NeuroImage, 180, 68-77. 10.1016/j.neuroimage.2017.06.061 Shows the standard deviation across folds strongly underestimates the real uncertainty of a cross-validated estimate, enough to cast doubt on published comparisons, with bars around ten points for a hundred samples. It justifies the decomposition measured here.
31. probabl (2026). skills, agent skills for scikit-learn, skrub and skore, BSD-3-Clause. github.com/probabl-ai/skills A suite chaining exploration, construction, smoke test, evaluation and audit, with an outer loop that sources the next experiment. It serves here as an example of a Ralph loop whose judge and stop conditions are written down, notably its decision tree for choosing a splitter.
32. Qiu, X. et al. (2024). TFB: Towards Comprehensive and Fair Benchmarking of Time Series Forecasting Methods. arxiv.org/abs/2403.20150 An open benchmarking protocol for time-series forecasting, built to avoid comparisons skewed by uneven tuning. It is the evaluation frame the project described here adopts.
33. Ansari, A. F. et al. (2024). Chronos: Learning the Language of Time Series. arxiv.org/abs/2403.07815 A foundation model that treats a numeric series as a sequence of tokens, the way a language model treats text. A candidate in the comparison, with no presumption of superiority.
34. Das, A. et al. (2024). A decoder-only foundation model for time-series forecasting. arxiv.org/abs/2310.10688 The other family of pretrained models in the comparison, trained on a large corpus of series to forecast without refitting. Its size makes it the project's CPU-feasibility test.
35. Angelopoulos, A. N., Bates, S., Fisch, A., Lei, L. and Schuster, T. (2022, revised 2025). Conformal Risk Control. arxiv.org/abs/2208.02814 Extends conformal prediction to risk notions broader than coverage rate, with a guarantee that holds on average under exchangeability. That is the assumption the dependence of financial series puts under strain, as the project sheet acknowledges.
36. Madaan, A., Tandon, N., Gupta, P., Hallinan, S., Gao, L., Wiegreffe, S. et al. (2023). Self-Refine: Iterative Refinement with Self-Feedback, NeurIPS. arxiv.org/abs/2303.17651 A loop where one model produces, critiques its own output, then corrects it, with no external signal. It predates the Ralph technique and establishes that critique-and-correct iteration is in no way specific to it.
37. Shinn, N., Cassano, F., Berman, E., Gopinath, A., Narasimhan, K. and Yao, S. (2023). Reflexion: Language Agents with Verbal Reinforcement Learning, NeurIPS. arxiv.org/abs/2303.11366 Keeps a memory of natural-language comments across attempts, standing in for a learning signal. A second antecedent showing that memory between turns also predates the Ralph technique.

Two trails to keep going. My favorite AI books, for the annotated survey. The sprezzature gallery, to see what a body of figures looks like once every one of them has been through the loop.