"While the counter is below ten, go round again." It is one of the oldest sentences in computing and it works for a reason nobody notices any more: the machine knows how to answer the question it is being asked. Comparing a counter to ten costs it one clock cycle, out of the billions a processor runs through every second. A billionth of a second, give or take, for an answer that cannot be wrong. The loop gets all its power from there: on every turn it asks permission to continue and the permission costs almost nothing.
Now try that with the questions that really decide the quality of a piece of work. Can anyone read this chart? Does this paragraph truly follow from the last one? Does this forecasting model deserve to be put to real use? A clock cycle answers none of them. So we pulled them out of our loops and pushed them to the very end, to a single review by hand, done by someone already tired.
The Ralph loop puts them back. It keeps the frame of the while loop and installs a judgment where the test used to be. The judgment comes from a human eye, a vision model that describes the image or a dataset held out on the side. One part changes and everything you can automate changes with it. A trap comes with it: keep asking the same judge and the loop ends up learning the judge's quirks instead of the subject. Statisticians call this overfitting. We are going to measure it and watch it happen on real data. From that we will draw two rules of conduct: a judge who is not the actor and a number of tries counted in advance.
The Ralph technique, on one line
The name comes from Geoffrey Huntley, an Australian engineer who described in July 2025 what he calls the Ralph technique1. His Ralph is Ralph Wiggum, the Simpsons schoolboy who misses the point with admirable constancy. And the whole technique types out on a single line, in the terminal where you give the computer written orders:
while :; do cat PROMPT.md | claude-code; done
Read it word for word: while true, hand the contents of the file PROMPT.md to the agent, then go again. You write down in a file what you want. A coding agent receives it over and over and nobody stops it. That agent is a language model, like the one behind ChatGPT, that has been allowed to read and write files and to run commands. You would expect a heap of incoherent files. What comes out is usable software.
How can that be? A language model keeps a limited amount of text in view during an exchange: its context window. The window is short and grows less reliable as it fills, so late in a long conversation the model loses track of what it decided itself. The project's files have no such weakness. On every turn of Huntley's loop, the agent starts from scratch, rereads the real state of the files, does one thing and writes it down. The work piles up on disk and the model's memory keeps none of it. Forgetting is what makes the technique work.
Huntley takes care not to sell it as a solution. He calls it "deterministically bad in a non-deterministic world." He asks for a brand-new project and an experienced engineer at the helm, then warns that the output comes out "undercooked, cooked or cooked with unspecified latent behaviours." He also reports a contract billed 50,000 dollars and delivered for 297 dollars of compute. That figure is his and nothing in this article verifies it.
Look closely at one detail of that line, because this whole article grows out of it. while : reads "while true": there is nothing in the condition. The original loop tests nothing and runs until a human stops it. Put something into that empty condition and the interesting part begins. In January 2026, Huntley himself argued that the idea holds for every task2, without saying how. Let us try to say how.
Four steps: act, render, judge, decide
Every variant you will meet reduces to four steps.
- Act: change one thing only, so you know afterward what to credit for the result.
- Render: turn that change into an object you can set in front of you, an image, a printed text, a table of measurements.
- Judge: examine that object.
- Decide: keep it or throw it out and note why.
The shared skeleton. From one loop to the next, only the rendered object and the identity of the judge change.
The second step is the one people skip, because it looks pointless. Skip it and you will judge an intention in place of a result: the loop will spin without learning anything. The fourth you pay for later. With no written reason the count gets lost and three weeks on, nobody knows how many ideas came before the one that survived. Is that count mere paperwork? The validity of everything else depends on it.
The idea has served before, under other names. Software engineers have called it the oracle problem for thirty years: when no specification tells you, how do you know whether a program is behaving well or badly? In 2015 a survey of the subject named the human as the last source of truth3. Machine learning got there through failure: alone with its mistakes, a language model corrects itself poorly and sometimes degrades in the attempt4. A July 2026 survey covers 1,250 papers and draws this from them: every improvement loop bets that some signal can replace human judgment5. A loop is worth what its signal is worth. So let us stop staring at the agent and turn to the judge it is given.
First testing ground: figures
This is 2026 and any model, you would think, should turn out a professional figure unaided. Yet figures still rank among the hardest objects to delegate: an image that came out of a machine is recognizable at first glance. That is why the loop makes its debut here, in that gap between promise and result.
With a sentence, you read it again and you know. With a chart, you don't: its code can be right from the first line to the last and the drawing wrong. The label wanders out of frame. The legend lands on the data. Two points fall on top of each other. A red-and-green palette turns to mud for anyone who cannot tell those two colors apart. Read the code a hundred times and none of these defects will appear: you have to look at the picture.
The Ralph Eyeball Loop6, at sprezzature.ai, turned that observation into a method: render, look, fix the source, repeat, as long as something is left to spot. Its rendering engine never calls on randomness: same source, same image. Then, figure after figure, the tool fills a critique log under eight required headings: composition, contrast, visual hierarchy, spacing, accessibility, color, an overall verdict and label legibility at the size it is rendered. Each pass adds to the log and erases nothing, so the history of what was seen and fixed stays. More than 150 kinds of charts in the sprezzature gallery have been through this loop.
Who looks? By default, the agent that just wrote the code judges its own output. As an option, a vision model installed locally does the first examination and a human rereads it. The gap between those two matters: a machine judge can be put under measurement. A benchmark published in September 2026 had models that read images as well as text judge charts. They show systematic weaknesses, first among them a strong tendency to accept everything7. The most lenient of all are the models rewarded, during training, when their answer pleased. Here is a judge bent by the very pressure it was supposed to referee. So a human eye stays in the loop.
The nearest example is right in front of you. This blog descends from a layout tool's export. Its stylesheet declared a spacing between paragraphs. Another rule canceled it. Across the eight articles then published, blocks of text were touching. The code was valid, the tests passed, the automated audits found nothing to say. A photograph of the rendered page, looked at by someone, did what nothing else had. That same pass flushed out two more defects of the same family: bibliography numbers perched on the first letter of their note and stray blank lines between list items. Three defects that one second of looking shows and no reading of the code ever will.
The loop comes into its own when the judge sees what the actor cannot. Take a person who tells colors apart normally. However much care they lavish on their figure, they will never see it the way a color-blind reader does, because that information is missing from their eye. So we ask a simulator to redraw the image as those readers perceive it, then in black and white. That is the harshest test of all: a figure legible in gray stays legible under every other color-vision deficiency. One second of computing is enough: the stopping condition costs almost as little as comparing a counter to ten.
Second testing ground: prose
Change the judge in the stopping condition and the same skeleton applies to text. The writing charter this article answers to makes it a rule8: you reread a text by pairs of paragraphs that follow one another, instead of paragraph by paragraph. The reason is simple: what catches a reader is at the seam between two paragraphs. So you read paragraph n then paragraph n+1 aloud, in one go, looking at the crossing from one to the other. Then you fix the text itself (the order of sentences, the last line of the first, the first line of the second) without settling for a note in the margin that "something sticks." Then the window moves on by one paragraph.
The windows overlap, so each paragraph takes the test twice, on arrival and on departure. The voice stands in for the render here. We think we know our text because we wrote it. Read aloud, it becomes a signal we receive instead of emit. This article got that treatment.
The danger of grading your own writing has been measured too. In 2024, Jane Pan, He He, Samuel Bowman and Shi Feng had one model write and grade in a loop: the grade climbs while quality, as real readers judge it, stalls or falls. And the more writer and grader share their context, the worse it gets9. You will not find a sharper argument for handing the reread to someone who did not write the text.
The voice has a blind spot all the same. It tries the seam between two paragraphs and says nothing about the inside of either. An ambiguous sentence from your own hand flows without a snag, since you already know what it means: memory plugs the hole before the ear can hear it. So we need a second instrument, on the same unit of text, looking elsewhere.
The second instrument goes by the name ABABA, which says what you do. A passage written in French goes into English, returns to French, sets off again. A stands for one language, B for the other. Count the letters: five states, four translations, two full round trips.
Let \(T_{AB}\) be the translation into English and \(T_{BA}\) the road back. One full round is their composition \(R = T_{BA} \circ T_{AB}\): first the outward leg, then the return. The versions then follow one another like this:
A well-written passage returns from the trip intact, because its words say its meaning without counting on the reader's goodwill. Mathematicians call it a fixed point of \(R\): the round no longer changes it. In machine learning the name is cycle consistency. That idea took a hundred years to get back to text, after a detour through photographs of horses.
It all starts with a writer's revenge. In 1903, Mark Twain published his jumping-frog story in three versions: his own, the French translation Thérèse Bentzon had made of it and his own word-for-word retranslation of that French back into English, French syntax and all10. The result is grotesque and that is the point: Twain shows what the trip did to his text. The device got serious in 1970, when Richard Brislin made it the standard quality control of cross-cultural research11. A questionnaire translated and translated back shows, through the gap, the badly worded question before it can skew a clinical study.
Enter the horses. In 2017, Jun-Yan Zhu, Taesung Park, Phillip Isola and Alexei Efros tried to change photographs of horses into zebras12. How do you teach that to a machine, when no matching pairs of photographs exist and so no answer key? Their method, CycleGAN, demands that a round trip give back the starting image. Where did they get the idea? In a talk, Efros gives the answer, which brings the story full circle13: Mark Twain and back-translation! The method needs no expertise. To get a text into a language you do not speak, you pay one translator to go out and another to come back, then you see whether the thing still stands up. Born in text, the technique had wandered off to images. We are taking it home.
In the same talk, Efros uses an image to explain what a round trip teaches. Picture a setup that compresses a piece of data then rebuilds it. The bottleneck in the middle here is a foreign language: the text has to squeeze through and come out the other side. It loses a lot on the way, the turns of phrase that are ours without meaning anything, the word order we had thought elegant. Above all, an ambiguity does not get through, because the translator has to pick one reading and stick to it.
When a sentence comes home deformed, the text was leaving its reader too much freedom. Is that an ambiguity, an idea left too vague, a turn of phrase the other language will not render, a usage peculiar to one language, a mere taste of the translator's? The loop points at the spot without ruling and going to look is our job.
Yet it sees what we no longer can. We already knew what we meant. The translator does not: it is the first naive reader at our disposal. An acronym never explained, a term defined nowhere, a pronoun whose referent we forgot three sentences back: none of the three survives the trip.
A stable text still proves nothing, as Efros says himself in the same talk. If all you ask is a return to the starting point, the neural network learning to transform the images can cheat by not moving at all: changing nothing cycles perfectly. A second demand had to join the first, that the transformed horse really look like a zebra. The same year, someone measured the cheat. Casey Chu, Andrey Zhmoginov and Mark Sandler showed that CycleGAN learns to tuck the starting image away inside invisible noise slipped into its own output14. The cycle closes perfectly and nothing has been translated. Its discoverers speak of steganography.
So the criterion has two parts, both required.
- Stability compares each version with the previous round: is the text still moving?
- Fidelity compares it with the original, always, never with the round before. In practice you write out what the starting text claims and what the arriving text claims, then compare the two lists.
Why the asymmetry? Because a drift in meaning advances by small steps, invisible one at a time and considerable once added up. And it covers its own tracks: by the second round the text has come to terms with its own error.
The trap announced at the start reappears here, very clearly: the loop learns its judge's quirks. With every round trip, the text is pulled toward one particular state, whatever its starting point: the translator's average prose. Iterating until the text stops moving therefore leads there, to flattened vocabulary, dead rhythm and bolted-on transitions. The loop clarifies and flattens together, without being able to separate the two. Something similar has been measured elsewhere. In 2024, Ilia Shumailov and his coauthors showed that a model retrained on its own outputs loses first whatever was rare in its answers15. The comparison has a limit: they retrain the model and we merely query it. Here the text grows poorer and the model stays as it was.
One ban protects us: never keep the loop's output. Keep whatever moves from one round to the next. A passage that refuses to settle is a badly written passage and that is the one you rewrite by hand. The signal is the instability, not the final text. If a round turns "X" into "Y," the useful information is that X could be rewritten, so a reader could have understood it as Y. We asked for nothing more. What follows gets written by hand and the style stays ours.
Alternating the translators costs little: Twain and Efros already did it by hiring two different people. With a single model, the text ends up talking like that model. With two models in alternation, a text that settles has to satisfy both sets of preferences, which pulls it away from either one's habits. That is an intuition and no proof backs it.
Choosing the language to pass through takes the same care. French and English resemble each other too closely. A turn of phrase copied word for word from one language into the other slips across unresisted and a badly built French sentence stays badly built in English without anything sounding wrong. To diagnose, pick a genuinely distant language, Japanese or Finnish: it forces a real re-encoding.
A serious objection comes from Harold Somers, who showed in 2005 that the round trip is a poor way to grade a machine translator16. The error going out and the error coming back offset each other, so a successful round says nothing about the quality of either translation. The objection is sound and it targets a different use. He varies the translator with the text fixed, since he wants to grade the translator. We do the reverse: with the translator fixed and assumed good, the text is what varies.
Used this way, the method holds, as two recent measurements confirm. In 2020, Jihyung Moon, Hyunchang Cho and Eunjeong Park showed that the round trip predicts quality well if you compare meanings and not words17. In April 2026, Ronald Skorobogat, Ameya Prabhu and Matthias Bethge measured a correlation of 0.94 (1 would be perfect agreement) between a round-trip score and the preferences of real users. Two established multilingual benchmarks drop to zero or go negative18. Somers remains right on one point: those offsetting errors forbid any conclusion from one round with one model.
That gives two instruments on the same unit of text: the ear tests the seam, translation tests the inside. You start with the voice, which is free and settles half the problems. You launch the round trip only on the pairs that resist and never more than two rounds on a passage that yields. After five rounds, what you are measuring is the translator's style and no longer the text.
The trap: a judge who can be bought
A loop optimizes what it measures. Put a judge into the stopping condition and the loop starts optimizing that judge. In 1975 Charles Goodhart saw it happen in British monetary policy: as soon as a statistical regularity serves as a target, it collapses19. The anthropologist Marilyn Strathern gave it in 1997, while studying university assessment, the wording that stuck: when a measure becomes a target, it ceases to be a good measure20. The modern version has been measured on language models. Leo Gao, John Schulman and Jacob Hilton pushed optimization harder and harder against an artificial judge. The score climbs without end. True quality rises, plateaus, then drops21. The judge bends as you push, without breaking at once: nothing is harder to notice.
In a Ralph loop, that corruption has a price you can compute. Suppose you are improving a forecasting model little by little. On each turn, the agent proposes a change and you measure it on a validation set: data held out on the side, never used for training, only for arbitration. That measurement is noisy: taken on a limited amount of data, it would come out differently on other data. Let \(\sigma\) be the typical size of those fluctuations. Each idea also has a real effect, sometimes good, sometimes bad, of typical size \(\tau\). Try \(K\) ideas and keep the one showing the best measurement.
Two hundred thousand simulated draws, with measurement noise about 3.3 times the spread of the true effects. At the top, what validation announces. At the bottom, what is left when you measure elsewhere.
In the simulation above, measurement noise is about 3.3 times larger than the gap between a good idea and a bad one. In practice that is nothing extravagant. Try eight hundred ideas: the winning change shows a gain of 3.32 on validation. Its true gain is 0.27. Twelve times less! Production always claws back that gap between the two curves.
That gap is no accident. Picture a competitive exam. Whoever takes the top mark generally has ability and luck, since picking the best paper also picks the luck. To estimate their true level, you do better to aim a little below the mark. That discount has a formula. For a measurement \(m\), the best estimate of the true effect \(\theta\) is not \(m\): you have to pull \(m\) toward zero and pull harder the noisier the measurement is:
Read it in words. On the left is what we want: the true effect, once the measurement is known. On the right is the measurement multiplied by a number below 1: the true effects' share of all the spread, noise included. With no noise, that ratio is 1 and you believe the measurement. When noise dominates, it falls toward zero and the measurement is worth almost nothing.
This calculation is called Bayesian shrinkage: it combines what you measured with what you already knew. It belongs to the family of estimators made famous by Charles Stein in 1956, then by James and Stein in 196122. In 2011 Bradley Efron connected it to selection bias23: a value chosen for being the largest always overstates what it measures. With the simulation's values the discount amounts to dividing by 12.1: whatever validation announces, divide it by twelve. Check it on the two curves: 3.32 divided by 12.1 does give 0.27.
The number of ideas tried leaves that factor untouched. It changes the size of the gain you think you read and that does not exist. That mirage grows like \(\sqrt{2\ln K}\): the logarithm gains a notch only when the number of ideas is multiplied and the square root slows that climb further. That function climbs very slowly and never stops. Let the loop run and in the long run it will announce as large a gain as you like on its judge, without having improved anything. That mechanism also explains why so many published results fail to replicate, as John Ioannidis showed in 200524. Oil economists had put a name on it by 1971: the winner of an auction is whoever most overestimated the tract, hence the winner's curse25.
This simulation assumes independent ideas with no average talent. That is pessimistic for a competent engineer, whose proposals are not drawn at random, and optimistic for an agent producing twenty variants of the same idea. It describes a mechanism and passes no verdict on any particular loop. Two findings survive those caveats: what you read exceeds what you get and the gap widens with the number of tries.
The trap, on real data
You may object that a simulation is only a simulation. So here is the same experiment on real data, harsher still than the model. Take eight hundred images of handwritten digits and cut them in two. Four hundred serve the judge, through cross-validation, explained further down. The remaining four hundred become a frozen test, invisible to the loop. On each turn, the loop proposes a small change to the input variables, accepts it if the validation score rises, rejects it otherwise. Of two hundred proposals examined, ten get accepted. The full protocol is given so it can be redone instead of believed. It includes the random seed, the number that pins down the randomness: rerun the program and you get the same experiment. The load_digits dataset from scikit-learn, random seed 20260922, logistic regression (a simple model that chooses between categories), five-fold cross-validation.
Each stair of the upper curve is a change accepted for having raised validation. The lower curve tells you what those ten decisions were really worth.
In ten steps, each justified by a measured rise, validation gains three points. The frozen test drops half of one. All that announced progress was noise the loop had chased. The disturbing part is that the loop failed the very way it would have succeeded: its validation curve climbs in steps, like the curve of a loop that is genuinely winning. Seen from the inside, the two situations look alike. To tell them apart you need a frozen set, consulted once.
A second objection weighs more than the first, because the largest empirical check points the other way. Rebecca Roelofs and her coauthors studied more than a hundred online prediction contests on the Kaggle platform. They set the public leaderboards, the product of thousands of submissions, against the private leaderboards revealed once at the end. They found little overfitting29. The explanation is called effective multiplicity: twenty variants of the same idea do not count as twenty independent tries. Our handwritten digits draw nearly independent tries: we chose the unfavorable case on purpose. So the question to ask is how many genuinely different tries a loop has made.
The budget makes a number believable
Should you then give up searching in a loop? A comparison says no. To tune a model, the classic reflex is the grid: you write a list of setting combinations ahead of time, evaluate them all and keep the best. In 2012, James Bergstra and Yoshua Bengio showed that at equal budget, combinations drawn at random already do better than a regular grid26. A Ralph loop takes one more step: what it proposes next depends on what the last measurement said. Statisticians call that adaptive analysis and rightly distrust it, because a validation set wears out much faster when the questions depend on the answers already obtained. For that case, Cynthia Dwork and her coauthors built quantified guarantees27. Everything therefore predicts a loop worse than the grid. The simulation says the opposite.
One judge, one evaluation budget and two ways of searching. The grid draws every candidate from scratch. The loop restarts each time from the previous winner.
At equal budget, the grid obtains a true gain of 0.27 and the loop 1.64: six times more. What explains the lead? A third experiment answers, in the manner of the control group of a drug trial. The same step-by-step walk, accepting everything without ever looking at the measurement, brings nothing. So the work comes from the selection, not the accumulation. Hand that walk instead to an oracle that could see the true effects and it would gain sixty times more. With its noisy judge, the loop recovers only two percent of what was available, roughly one part in sixty. The mirage is identical on both sides: grid and loop overstate their gain by as much. The inflation depends only on how many times you consulted the judge, however you searched.
The budget rule that follows is not mine. In 2014, David Bailey, Jonathan Borwein, Marcos López de Prado and Qiji Zhu treated this very problem for financial backtests28. A backtest replays an investment strategy on the past. Below a certain amount of data, they established, a flattering track record proves nothing. Five years of history allow no more than forty-five different configurations. For a loop, set a tolerance \(\delta\) in advance: how many times the noise \(\sigma\) you accept the announced gain to exceed the real one. The same bound then gives a cautious rule:
In words: the number of tries allowed stays under the exponential of half the square of that tolerance. Take a tolerance of 3: the square is 9, half of it 4.5, the exponential about ninety. The exact calculation would allow more: about three for a tolerance of 1, twenty-seven for 2, four hundred and forty-three for 3. The formula concedes only one, seven and ninety, so it is cautious. And \(K\) counts genuinely different tries, not loop turns.
One point survives every caveat, proven since 2014 in an entirely different field: limiting the number of turns is worth far more than the compute it saves. Without that limit, the final number stops meaning anything.
The loop earns its place through something other than that factor of six: a grid assumes you can write the list of candidates in advance. Adding a derived variable, changing what the model is trying to reduce, removing an answer slipped by mistake into the inputs: those changes do not come as a list. You cannot put them in a grid, because you cannot write its axes. The loop goes looking where no grid can be drawn. That is its irreplaceable advantage. Everything else comes at a price.
The judge you meet every day: cross-validation
All of the above depends on \(\sigma\), the judge's noise. And in a real project that noise gets set behind your back, on the day you choose how to cut up the data. The usual tool goes by the name of cross-validation. Cut the dataset into \(k\) parts, which we call folds. Train on all the folds but one and measure on the one left over. Do that \(k\) times, changing which fold sits out. Finally announce the mean of the \(k\) scores, with an error bar meant to say what that mean is worth.
That bar nearly always flatters, for a simple reason: the calculation pretends the \(k\) measurements are independent, when two folds share nearly all their training data. Gaël Varoquaux put numbers on the problem: on small samples, published bars are routinely twice too narrow30. On our handwritten digits, the variability that matters, from one sample to the next, comes to nearly double the one quoted between folds. Underestimating \(\sigma\) by a factor of two means believing in four times more progress than there is, since noise enters the shrinkage formula squared. The split therefore sets the exchange rate between what you read and what you get.
That is what the tool suite published by probabl around scikit-learn, skrub and skore codifies31. It splits by groups when several rows come from the same subject and by time, with a buffer zone, when time order matters. It also warns against the habits that produce overconfident bars. You can recognize a Ralph loop written out in full: a logbook where state lives, a judge that is a report separate from the pipeline's author, a written reason with every proposal. Several steps stop dead instead of carrying on over a missing dependency.
One detail in that suite deserves copying everywhere. Its very first check, which merely confirms that everything runs, rests on an exact count: the number of predictions must equal the number of rows to predict. That check catches one precise and frequent fault. A question that needed judgment becomes a free test. Every Ralph loop gains from manufacturing as many of these as it can, because whatever the machine can settle has no business in the expensive condition.
A project under way applies these rules to forecasting financial series. Two students of the specialized master's in artificial intelligence at CentraleSupélec are carrying it out, on a proposal by Marianne Clausel, of the Université de Lorraine, Georges Oppenheim and myself. Its evaluation protocol is called LookForward, which others call walk-forward. On each date \(t\), it computes a statistic \(S_t\) about the steps that follow \(t\): the yellow zone in the figure. That computation reads only the blue dates, those whose information was already published, so an indicator published on the fifteenth never feeds a forecast made on the tenth. The quality of \(S_t\) is judged later, on the day the yellow values become known.
The blue takes two forms, both of them LookForward. The expanding window starts at the beginning of the series and grows with \(t\): it reads every date \(s\) from \(1\) to \(t\). The rolling window keeps a fixed width \(r\) and moves with \(t\): it reads the dates from \(t-r+1\) to \(t\). Then the date moves forward one notch for the next computation. Which dates \(t_{\min}\) and \(t_{\max}\) bound \(t\)? That is another question, still open.
LookForward with an expanding window (top) and a rolling window of width r (bottom). Here the judge is a discipline of the calendar and not a randomly drawn dataset: no row ever reads to the right of its date t.
In this project, a Ralph loop under budget tests each change with LookForward and writes down why it accepts or rejects it. Set apart, frozen, opened once, a final chronological test yields the number that will be published. The project compares classical statistical methods, machine-learning methods and time-series foundation models, following the open protocol TFB32, with Chronos and TimesFM among the candidates33, 34. It also allows itself to abstain, backed by conformal risk control35, whose guarantee rests on an assumption financial series strain.
| Loop | What gets rendered | Who judges | How the judge gets bought | The guard |
|---|---|---|---|---|
| Code | The repository after one turn | The test suite, then a human | Tests become the target and code passes without being right | Tests written first, by someone else |
| Figures | An image, always the same for a given source | An eye or a vision model | You get used to your own figure and stop seeing it | Grayscale, color-vision simulation, a fresh pair of eyes |
| Prose | The text read aloud, then translated and translated back | The ear for the seam, a translator for the inside | The text converges on the translator's average prose | Keep what changes between rounds and not the output, alternate translators |
| Forecasting | Dated errors, horizon by horizon | A rolling validation | The more you consult a validation set, the more it wears out | A budget, a recorded reason, a frozen test |
Huntley's loop on code, then this article's three testing grounds: figures, prose, forecasting. The fourth column says the same thing four times, in each ground's own language: the judge becomes the target.
Where the idea comes from and what this article adds
Let us set the record straight, because the word is starting to mean everything. Chaining production, critique and correction is in no way specific to Ralph, which came after. Self-Refine, published in 2023, has the model critique itself36. Reflexion, the same year, keeps a memory of comments from one attempt to the next37. Huntley's technique stands apart through a combination.
- A fresh context on every turn, instead of a lengthening conversation.
- State kept outside the model.
- The code produced, actually executed.
- One thing done on each turn.
As for the loops on prose, figures and forecasting described above, they are my generalization and not an established sense of the term. Search for "Ralph loop" elsewhere and you will find code and nothing else.
The numbered part comes from elsewhere too. In 2014, quantitative finance proved that choosing among \(N\) backtests imposes a history of minimum length28. In 2015, theoretical computer science proved that a held-out set withstands only a limited number of adaptive questions27. Between 2022 and 2024, the researchers who want an artificial intelligence to pursue what we actually ask of it measured that pushing too hard against a stand-in judge destroys real quality21. Shrinkage toward the mean goes back to the 1950s. And meanwhile, in 2025, software engineering put into production a loop with no stopping condition at all. As far as I know, these literatures rarely cite one another. This article's only contribution is to say that they describe the same object, with the budget rule holding for the programs that run an agent in a loop as it does for market strategies.
The guards
Before launching anything, check these guards.
- State is kept outside the model, in files and their history, never in a conversation.
- The judge is independent of the actor: the eye that evaluates is not the hand that drew, the validation set is not the training set.
- A budget limits the number of turns.
- Every turn leaves a written reason: without it the count of tries gets lost and the final gain becomes uncheckable.
- Abstention is a permitted exit, on the same footing as success: a loop that is not allowed to conclude it found nothing will always find something.
Drop a single guard and the final number is no longer checkable: the loop becomes a "retry until it passes" again.
All of them obey one principle, which Efros states better than I can in the talk already cited13. An incorruptible judge does not exist. The data is finite, so there will always be some path for cheating. Plugging one hole opens another. All you can do is arrange the problem so that cheating costs more than working. Efros offers the image of a well-run country: you are free to break the law there, you just gain nothing by it. In a badly run country, cheating is the rational calculation and everyone cheats.
He shows it in his own classroom. A semester of work costs a lot. Memorizing last year's past papers is cheap and often enough to pass the exam. The student who makes that calculation is rational, not dishonest: the exam is what is badly built. A validation set consulted two hundred times is those past papers. The budget, the written reason and the frozen test do not moralize at the loop. They make cheating more expensive than working.
Where does the analogy with the while loop give way? Let a classic loop run forever and you see it at once: the machine heats up and nothing comes out. Let a Ralph loop run forever and every turn produces plausible work, a slightly different figure, a slightly reworked paragraph, a slightly better score. It looks like progress, not like a breakdown. That is what makes it dangerous.
So before your next loop: render what you just made, actually look at it, fix the source and not the image, count your turns. Three of those steps cost seconds. The fourth, the one everyone forgets, alone decides what the other three are worth. The agent's talent counts for less than the evaluation around it, provided the agent cannot overfit it.
Let's talk AI.
The figures in this article were made with sprezzature.ai by Deraison.ai
Bibliography
Every title below went into writing one particular section: the note says which. None is cited second-hand.
- Huntley, G. (2025). Ralph Wiggum as a "software engineer". ghuntley.com/ralph The page that names the technique and gives its shell line. The author also states the caveats quoted here: greenfield only, a senior engineer required, output whose doneness is not guaranteed. It carries no stopping condition at all: the operator decides. The cost figure cited is his.
- Huntley, G. (2026). everything is a ralph loop. ghuntley.com/loop Argues the pattern holds for all tasks and sketches several orchestration variants. It is the most direct antecedent of the generalization defended here; generality is asserted rather than structured, with no common skeleton and no failure-mode analysis.
- Barr, E. T., Harman, M., McMinn, P., Shahbaz, M. and Yoo, S. (2015). "The Oracle Problem in Software Testing: A Survey," IEEE Transactions on Software Engineering, 41(5), 507-525. 10.1109/TSE.2014.2372785 Surveys thirty years of attempts to decide automatically whether observed behavior is correct and concludes that where specifications, models and metamorphic testing stop, the last source of oracle information remains the human. This is software engineering's statement of the problem, long predating agents.
- Huang, J., Chen, X., Mishra, S., Zheng, H. S., Yu, A. W., Song, X. and Zhou, D. (2024). Large Language Models Cannot Self-Correct Reasoning Yet, ICLR. arxiv.org/abs/2310.01798 Separates self-correction from the model's own resources alone from correction leaning on an outside signal and shows the former fails, performance sometimes degrading after correction. It is the empirical case for the judge coming from somewhere other than the actor.
- Chen, M., Wang, L. and Qu, B. (2026). Recursive Self-Improvement in AI: From Bounded Self-Refinement to Autonomous Research Loops. arxiv.org/abs/2607.07663 A survey of 1,250 papers holding that every improvement loop is a claim that some signal can substitute for human judgment and ranking those signals from formal verifier to intrinsic self-assessment. It is the closest statement to this article's framing and it came first of the two by several months.
- Harchaoui, W. The Ralph Eyeball Loop, Sprezzature. sprezzature.ai/ralph-eyeball-loop.html Applies the loop to visual artifacts and separates three complementary passes: make it from code, audit the source, look at the render. It is the source of the four steps, the eight critique headings and the argument for grayscale as the severest test.
- Wu, L., Zhao, H. H., Zhang, Z., Tang, J., Wu, J. and Wang, A. J. (2026). ChartJudgeBench: Evaluating LMM Judges for Chart-to-Code Generation. arxiv.org/abs/2609.24210 Evaluates multimodal models used as chart judges and documents four systematic weaknesses, among them a marked tendency to accept and greater leniency in reinforcement-trained models. It is the measurement of the visual judge the figure loop depends on.
- Harchaoui, W. WRITING.md, a writing charter for English prose. gist.github.com Section 10 carries the loop over to prose: rereading by pairs of consecutive paragraphs, overlapping sliding windows, fixing the text rather than the impression. It is the charter this article applies to itself.
- Pan, J., He, H., Bowman, S. R. and Feng, S. (2024). Spontaneous Reward Hacking in Iterative Self-Refinement. arxiv.org/abs/2407.04549 Shows that in a loop where one model both writes and grades, the grade rises while quality as perceived by real readers stalls or falls, the more so when writer and grader share context. The failure mode described here was therefore already named and measured on prose.
- Twain, M. (1903). The Jumping Frog: In English, Then in French, Then Clawed Back into a Civilized Language Once More by Patient, Unremunerated Toil, Harper & Brothers. Twain prints his story, the French translation Thérèse Bentzon made of it and his own word-for-word retranslation, side by side. The demonstration is comic and exact: the round trip makes visible whatever was fragile in the text. It is the literary ancestor of the loop described here.
- Brislin, R. W. (1970). "Back-Translation for Cross-Cultural Research", Journal of Cross-Cultural Psychology, 1(3), 185-216. 10.1177/135910457000100301 Establishes back-translation as quality control in cross-cultural research; it has been the standard procedure ever since for translating a clinical or psychometric questionnaire, where one lost nuance skews a whole study. It settles that the return gap is an exploitable signal rather than noise.
- Zhu, J.-Y., Park, T., Isola, P. and Efros, A. A. (2017). "Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks", ICCV, 2242-2251. arXiv:1703.10593 CycleGAN: with no matching image pairs available, the training constraint becomes the round trip itself. It is the formalization of cycle consistency used here, together with the proof that it can stand in for a missing answer key.
- Efros, A. A. (2018). Self-supervision, Meta-supervision, Curiosity: Making Computers Study Harder, lecture at CIIRC, Czech Technical University in Prague, 25 May 2018. youtube.com/watch?v=_V-WpE8cmpc Efros explicitly credits CycleGAN's idea to Mark Twain and back-translation and describes the setup as an autoencoder whose narrow passage is another language (at 48 min). He also states the limit taken up here: without a second constraint, the network cheats by not moving. And the formulation that carries the end of this article: make cheating more expensive than doing the work.
- Chu, C., Zhmoginov, A. and Sandler, M. (2017). CycleGAN, a Master of Steganography, "Machine Deception" workshop, NIPS. arXiv:1712.02950 Shows that CycleGAN conceals the source image in an imperceptible high-frequency noise, which lets it reconstruct exactly without having translated anything. It is the demonstration that cycle consistency alone is degenerate. Hence fidelity is measured here against the original, never against the previous round.
- Shumailov, I., Shumaylov, Z., Zhao, Y., Papernot, N., Anderson, R. and Gal, Y. (2024). "AI models collapse when trained on recursively generated data", Nature, 631, 755-759. 10.1038/s41586-024-07566-y Measures what becomes of a model retrained on its own outputs: the tails of the distribution go first, which is to say whatever was rare. Cited here as a measured analog of the attractor, with its limit stated plainly: theirs is a retraining loop, not a querying loop.
- Somers, H. (2005). "Round-trip Translation: What Is It Good For?", Australasian Language Technology Workshop, 127-133. aclanthology.org/U05-1019 The substantive objection: the round trip grades a machine translator very poorly, since errors on the outward and return legs cancel. Taken up here rather than set aside; it does not target the same use, but it directly motivates alternating translators.
- Moon, J., Cho, H. and Park, E. L. (2020). "Revisiting Round-Trip Translation for Quality Estimation", EAMT. arXiv:2004.13937 Rehabilitates the round trip by changing what gets compared: meaning embeddings instead of word overlap. It is the technical answer to Somers, hence the justification for the fidelity criterion used here, which bears on claims and not on form.
- Skorobogat, R., Prabhu, A. and Bethge, M. (2026). Round-Trip Translation Reveals What Frontier Multilingual Benchmarks Miss, 14 April 2026. arXiv:2604.12911 Measures a correlation of 0.94 between a round-trip score and user preferences, where two established multilingual benchmarks fall to −0.09 and −0.26. It is the most recent evidence that the round trip catches something real. It bears on models rather than on text, so it supports the use made here without proving it.
- Goodhart, C. A. E. (1975). "Problems of Monetary Management: The U.K. Experience," Papers in Monetary Economics, Reserve Bank of Australia. The original observation, made on British monetary aggregates: a statistical regularity stops holding once it is used as a policy target. It grounds the section on the corruption of the judge.
- Strathern, M. (1997). "'Improving ratings': audit in the British University system," European Review, 5(3), 305-321. Supplies the short formulation that became proverbial, out of ethnographic fieldwork on university assessment. Cited for the exact wording rather than through a secondhand quotation.
- Gao, L., Schulman, J. and Hilton, J. (2023). "Scaling Laws for Reward Model Overoptimization," ICML, PMLR 202, 10835-10866. arxiv.org/abs/2210.10760 Has a reward model stand in for the human judge and measures what happens as you optimize against it: the score climbs, true quality plateaus then falls, along a fitted functional form. Goodhart's law made quantitative, on a subject this article does not otherwise treat.
- Efron, B. and Morris, C. (1977). "Stein's Paradox in Statistics," Scientific American, 236(5), 119-127. An accessible account of shrinkage toward the mean: a value picked because it was the largest overstates what it measures. How much to take back depends on the ratio of noise to real spread. That is the boxed formula in this article. It dates from the 1950s.
- Efron, B. (2011). "Tweedie's Formula and Selection Bias," Journal of the American Statistical Association, 106(496), 1602-1614. 10.1198/jasa.2011.tm11181 Ties shrinkage explicitly to selection bias: the largest few estimates overstate their parameters, with the correction written from the observed marginal density. It is the exact address of the factor used here.
- Ioannidis, J. P. A. (2005). "Why Most Published Research Findings Are False," PLoS Medicine, 2(8), e124. 10.1371/journal.pmed.0020124 Makes quantitative the link between the number of hypotheses explored and the probability that a reported result is false. Same mechanism as the one simulated here, applied to the scientific literature.
- Capen, E. C., Clapp, R. V. and Campbell, W. M. (1971). "Competitive Bidding in High-Risk Situations," Journal of Petroleum Technology, 23(6), 641-653. Origin of the phrase "winner's curse," on auctions for oil tracts: the winner is the bidder whose estimate was highest, hence most wrong. The same structure as selection on a validation set.
- Bergstra, J. and Bengio, Y. (2012). "Random Search for Hyper-Parameter Optimization," Journal of Machine Learning Research, 13, 281-305. jmlr.org/papers/v13/bergstra12a Shows that at equal budget, drawing settings at random beats a regular grid, because performance depends strongly on only a few settings and a grid repeats its trials on the others. It is the comparison baseline for the model-selection section.
- Dwork, C., Feldman, V., Hardt, M., Pitassi, T., Reingold, O. and Roth, A. (2015). "The reusable holdout: Preserving validity in adaptive data analysis," Science, 349(6248), 636-638. 10.1126/science.aaa9375 Separates questions posed in advance from questions depending on answers already obtained and builds a mechanism restoring quantified validity to a holdout under adaptive querying, at the cost of an explicit budget of disagreements. It is the theoretical-computer-science proof that the budget is a necessity, not a prudence.
- Bailey, D. H., Borwein, J. M., López de Prado, M. and Zhu, Q. J. (2014). "Pseudo-Mathematics and Financial Charlatanism: The Effects of Backtest Overfitting on Out-of-Sample Performance," Notices of the American Mathematical Society, 61(5), 458-471. 10.1090/noti1105 Computes the expected maximum of \(N\) skill-free trials, bounds its growth by \(\sqrt{2\ln N}\) and derives a minimum-backtest-length theorem. It is the source of this article's budget rule, which only carries it across; the precedence is total, the bound itself coming from extreme-value theory.
- Roelofs, R., Shankar, V., Recht, B., Fridovich-Keil, S., Hardt, M., Miller, J. and Schmidt, L. (2019). A Meta-Analysis of Overfitting in Machine Learning, NeurIPS. Compares, across more than a hundred Kaggle competitions, public leaderboards built from thousands of submissions against private leaderboards revealed once: the observed overfitting is small, if anything. It is the empirical counter-evidence to the worst-case theory and the reason to speak of effective multiplicity rather than of turn count.
- Varoquaux, G. (2018). "Cross-validation failure: Small sample sizes lead to large error bars," NeuroImage, 180, 68-77. 10.1016/j.neuroimage.2017.06.061 Shows the standard deviation across folds strongly underestimates the real uncertainty of a cross-validated estimate, enough to cast doubt on published comparisons, with bars around ten points for a hundred samples. It justifies the decomposition measured here.
- probabl (2026). skills, agent skills for scikit-learn, skrub and skore, BSD-3-Clause. github.com/probabl-ai/skills A suite chaining exploration, construction, smoke test, evaluation and audit, with an outer loop that sources the next experiment. It serves here as an example of a Ralph loop whose judge and stop conditions are written down, notably its decision tree for choosing a splitter.
- Qiu, X. et al. (2024). TFB: Towards Comprehensive and Fair Benchmarking of Time Series Forecasting Methods. arxiv.org/abs/2403.20150 An open benchmarking protocol for time-series forecasting, built to avoid comparisons skewed by uneven tuning. It is the evaluation frame the project described here adopts.
- Ansari, A. F. et al. (2024). Chronos: Learning the Language of Time Series. arxiv.org/abs/2403.07815 A foundation model that treats a numeric series as a sequence of tokens, the way a language model treats text. A candidate in the comparison, with no presumption of superiority.
- Das, A. et al. (2024). A decoder-only foundation model for time-series forecasting. arxiv.org/abs/2310.10688 The other family of pretrained models in the comparison, trained on a large corpus of series to forecast without refitting. Its size makes it the project's CPU-feasibility test.
- Angelopoulos, A. N., Bates, S., Fisch, A., Lei, L. and Schuster, T. (2022, revised 2025). Conformal Risk Control. arxiv.org/abs/2208.02814 Extends conformal prediction to risk notions broader than coverage rate, with a guarantee that holds on average under exchangeability. That is the assumption the dependence of financial series puts under strain, as the project sheet acknowledges.
- Madaan, A., Tandon, N., Gupta, P., Hallinan, S., Gao, L., Wiegreffe, S. et al. (2023). Self-Refine: Iterative Refinement with Self-Feedback, NeurIPS. arxiv.org/abs/2303.17651 A loop where one model produces, critiques its own output, then corrects it, with no external signal. It predates the Ralph technique and establishes that critique-and-correct iteration is in no way specific to it.
- Shinn, N., Cassano, F., Berman, E., Gopinath, A., Narasimhan, K. and Yao, S. (2023). Reflexion: Language Agents with Verbal Reinforcement Learning, NeurIPS. arxiv.org/abs/2303.11366 Keeps a memory of natural-language comments across attempts, standing in for a learning signal. A second antecedent showing that memory between turns also predates the Ralph technique.
Two trails to keep going: My favorite AI books, for the annotated survey, and the sprezzature gallery, to see what a body of figures looks like once every one of them has been through the loop.