In short
For ten years I have run enterprise artificial intelligence projects through the same sieve: three gates, taken in this order. Useful: is the problem real and priced? Usable: does the tool survive real hands? Used: do people still come back to it in week three? Each gate fails in its own way and I have watched all three do so in the field.
The questions to ask, gate by gate, close the essay in the Monday-morning compass. The contents list just below lets you pick your own path.
A kite and its string
What keeps a kite from climbing? Its string, says common sense. Cut it and watch: the kite falls.
André Gide threw that image at his audience on March 25, 1904, in Brussels, at the Libre Esthétique. That avant-garde salon hung close to two hundred canvases by Manet, Monet, Degas, Renoir and Van Gogh that year1. Four lectures accompanied the show. Poetry, painting and music were already spoken for, so theater fell to him. Gide had wounds to speak from. His play Le Roi Candaule had survived a single performance in 1901 and his Saül had been waiting six years for a director brave enough to stage it2.
The theater of his day, he argued that evening, was dying of its own ease and only fresh constraints could save it. Believing that art rises higher the freer it is, he said, "is believing that what keeps the kite from climbing is its string." Then came the evidence. Michelangelo, so the story goes, owed the gathered pose of his Moses to a flaw in the marble. Aeschylus owed the silence of Prometheus to the small number of actors allowed on stage. Greece went further and banished the man who added a string to the lyre. "The great artist is the one whom hindrance exalts, for whom the obstacle becomes a springboard." And then the sentence:
“Art is born of constraint, lives on struggle, dies of freedom.”
Gide forged it by transposing Leonardo da Vinci, who wrote of mechanical force that it "is born of violence and dies of freedom"3. Constraint took the place of violence, art the place of physics. A line of mechanics had turned into a line about art.
My trade is applied mathematics, done with a computer, inside companies. People call that artificial intelligence now. I prefer the long version, because every word in it earns its keep. Mathematics supplies the rigor: a model is a sentence you can prove wrong. The computer supplies the reality of computation, data that arrives dirty, machines that cost money, response times you can clock. The company supplies the reality of people, budgets, customers, teams with no time to spare.
For more than ten years I have helped companies build these systems from prototype to production. I have built them for credit card fraud, lab analysis of medical samples, shorter eye exams, demand forecasting, video and music at industrial scale, then agents. This whole essay plays out between the prototype and production, so let us look at both up close. The prototype is the mock-up that impresses in a meeting: it works on ten chosen examples, on a Wednesday, in front of friendly faces. Production looks like the same system, except that now you have to live with it. There is real data, breakdowns, budgets, moods, teams who never asked you for anything, Friday at six included.
My clients' trades are languages. I learn each one and they learn mine. Somewhere in that back and forth, Gide's sentence became the working definition of the craft I love.
The constraint is whatever problem has to be solved: the real definition of success, the number you promise to move. I call that useful, because value comes from a constraint resolved. A project that resolves none has no reason to exist.
The struggle pits beautiful theory against the reality of the field: the interface, the integration, the trust. I call that usable, because the tool gets forged in that fight with the real. There are field problems to solve and habits to shake without breaking them, so that innovation pays off in the hands of the people doing the work.
Freedom, to me, tastes like work accomplished: the handover, a system that runs without me, the right to move on to the next project. I call that used, because the proof gets harvested along with that freedom. A use turned routine sets everyone free, the teams first, the practitioner next.
Gide's sentence, transposed to the craft: three words, three Us.
Useful, usable, used: three adjectives for one sieve, the one that separates a project that lasts from a demonstration that shines for a day.
A sieve with no inventor
The three Us predate me. As far as I know nobody can claim them, which is rather a good sign: the frameworks that serve belong to no one.
The foundations are more than thirty years old. In 1989 Fred Davis showed that two perceptions predict whether a technology gets adopted, its usefulness first, its ease of use second, the former weighing more than the latter4. Jakob Nielsen broke down what makes a system worth adopting: utility (does it do the right thing?) multiplied by usability (can people operate it?)5. The ISO 9241-11 standard spells usability out in black and white as effectiveness, efficiency and satisfaction, for given users in a given context6. Researchers writing in French settled on their own three words: utility, usability, acceptability7. Then practitioners wore the words down by using them. Useful, usable, used passed into the language of designers8 and health research now asks for results that are "useful, usable and used"9.
The bias of this sieve goes by the name of the third U. The university predicts acceptability, the practitioner observes use. Acceptable on paper earns me nothing. I get paid so that someone, on an ordinary Monday morning, opens the thing without giving it a thought. Out of that came three field-forged mottos:
- Relevance inside systems beats slogans. Usefulness is not proclaimed, it is read in the numbers of a running system.
- Integration into work habits beats the demonstration. The finest demonstration in the world weighs less than a tool that slips into Wednesday-morning routines, into software already open.
- Retention beats volume. First-month user counts are a vanity. Retention counts who comes back in the third month, once curiosity has faded and only the service rendered remains.
Why a sieve? Because artificial intelligence projects fail in bulk. The figures that circulate about this deserve their own error bars and three of them fall over at the first check:
- The famous "85% of AI projects fail" comes from a 2018 Gartner press release that predicted something else entirely, erroneous outcomes caused by bias, by 2022. Nobody has evaluated that prediction since10.
- The "87%" traces back to a sponsored article that cites no primary source11.
- MIT's "95% of generative AI pilots with no measurable return" calls itself "directionally accurate," an elegant way of asking for leniency12.
Then comes RAND Corporation, the source most quoted and least read. Right at the start of its 2024 report, a clause lies in wait and everyone skips it: "by some estimates," more than 80% of AI projects fail. Three words change the number's status, since the clause borrows it instead of measuring it. RAND's own contribution is sixty-five interviews with seasoned practitioners across government and industry. Those interviews go back to causes instead of the rate and the leading cause they point to is a misunderstanding of the problem to solve13. So the lesson about causes is firsthand and the percentage secondhand. I have no other number to put in its place.
Keep the hierarchy of causes: projects fail on the problem (useful), in the field (usable), on adoption (used). The sieve claims nothing more. In through the first gate, then.
The three-U sieve, from prototype to production: each gate has its failure mode.
Useful: the constraint
When doctors suspect a heart attack, they measure a protein in the blood called troponin: the higher it climbs, the more the heart is suffering. Hand that risk estimate to a model. It has two ways of being wrong. If it aims too high, a patient occupies a bed for nothing. If it aims too low, that patient goes home, where they may die of it. The error is the same size in both cases. Only its sign changes and the consequence flips. An artificial intelligence system that ignores this asymmetry has chosen, by default, to treat both errors the same. That symmetric default looks like the absence of a choice: it is a bad choice made in silence.
It all starts well before the first model, the instant a need turns into an idea. Long before anyone says "wouldn't it be great," an artificial intelligence idea starts with someone who has a problem. The wow effect stands in front of that problem and my job is to go find what it covers. So I follow up with a question that sounds trivial: what is the unit? There are five: money, time, energy, care and the impossible made possible. Time counts minutes handed back to a nurse or a factory worker. Energy counts computations, trips, waste avoided. Care counts medical errors dodged, discomfort reduced. The impossible is the one exotic currency in the set, covering what was not done at all: reading every customer review, listening to every minute of airtime, describing every image of an entire catalog. If your project pays in none of these currencies, you are holding a toy. And the silence that follows my question is already a data point.
To get from the unit to the number, I need a tool everyone can read: the confusion matrix. The name intimidates, the object is simple. Take a binary decision, fraud or no fraud, then cross two questions: what did the system say? what was actually true? Two possible answers to each question make four outcomes, not one more:
- The system flags a real fraud: a good catch (engineers say true positive). The alarm rang at the right time.
- It accuses an honest customer: a false alarm (false positive). The alarm rang for nothing.
- It lets a fraud through: a miss (false negative). The alarm stayed silent at the worst time.
- It leaves an honest customer in peace: a correct rejection (true negative), the silence of good days.
Every decision model, from the simplest to the deepest, ends up in these four boxes. Count what falls into each over a year and you already know it.
The confusion matrix: two questions crossed, four outcomes.
The method then takes one move: lay over that count the matrix of business consequences, priced in the client's unit. The right indicator, the number an executive looks at to decide, is born in the business, never in a laboratory. It measures what the system brings in, net of its errors, in the currency of the business.
Engineers have their internal dials for this and I introduce them to my MBA students with two Avengers. Precision is the archer Hawkeye: when he shoots, he hits. It answers one question: when the system cries fraud, is it right? Recall is the colossus Hulk: nothing that needs destroying escapes him. It answers another: out of a hundred real frauds, how many does the system catch? The archer rarely misses but lets things through. The colossus spares nothing and breaks the furniture along the way. Neither is the right hero in all circumstances, since it all depends on the price of the furniture and the price of the missed target. Combine Hulk's recall with Hawkeye's precision and you have the perfect artificial intelligence, the one that catches everything and hits nothing but its targets. It does not exist. My job is to trade one against the other, at the business's price. These dials are for tuning the machine. They say nothing to a board of directors.
From a slide of my MBA course: Hawkeye for precision, Hulk for recall. Characters © Marvel.
In a boardroom, only the translation into business currency moves anyone. In ophthalmology, a client wanted to shorten eye exams and success was not measured in "accuracy": an exam went from four minutes ten to two minutes seven, at equal prescription. Two hundred and fifty seconds became one hundred and twenty-seven, nearly half saved! We got there by punishing the machine harder for the errors that cost the patient discomfort, which is what an asymmetric loss function does. Measured, not extrapolated.
Every box has its own price, set by the size of the order: the asymmetry of errors is the message.
A full case belongs here, because a sieve gets judged on a day of decision. I spent six years at an auto-parts e-commerce company where credit card fraud detection had been running for years and nobody was complaining: the model posted good scores, the team tracked them weekly, the dashboards stayed green. Customer service, though, kept reporting a friction the scores never saw: honest customers refused at checkout, who called once and then stopped coming back.
No dashboard ever offers you the hard choice we made: to accept letting more fraud through. Said like that, it is indefensible. Put in the currency of the business, it is addition. A fraud that goes through costs the order, the dispute fees and the shipping. A customer wrongly refused costs the margin on that order, then every order they will place with a competitor.
So the model stayed as it was and we rewrote the definition of error. Two corrections were enough.
First, the price of an error varies with the order. Wrongly refusing a set of brake pads and letting a fraud through on a complete gearbox are faults of very different weight, yet the confusion matrix files them in the same box. We replaced the two fixed prices with two functions of the amount: the price of an error is now worked out from the size of the order. The system now minimizes that matrix, which changes with every order. A model of this kind always has a threshold you tighten or loosen. Setting it to the best cost instead of the best score is a first improvement, feeding the cost into the training itself a second. The engineer's standing dilemma between precision and recall then becomes a management decision the leadership can understand, contest, then own.
Second, a regret took the place of the score. Regret means the gap between what the decisions cost and what the best available decisions would have cost over the same period. A score invites contemplation. A regret gets compared to zero, then to the regret of the dumbest policy you could hold, accept everything or refuse everything. The dumb policy becomes the bar to clear. Below it, a system has no business in production, whatever its scores.
Look at what came of it: the model's precision went down while revenue net of losses went up. In any technical review, that was a failure! The gain concentrated where the money is, on the large orders, because those weigh heaviest in the new definition of error. The argument in committee moved ground: it now aimed at the two functions of the amount, meaning commercial policy, which was the right room for it.
Does the intuition hold outside that client? The measurement is under way on a public set of card transactions, in work still under peer review. The data are cut in time order, so the system gets judged on months it has never seen. On those data, regret falls, most of all for the large amounts.
Those two functions of the amount hide a commercial decision inside a parameter. Nobody stands by to check that they are right. An error in writing them goes straight into what the system learns, with no review in between. So we reread them twice a year with finance and product, like a price list, never like a physical constant. Add a cruel delay: a chargeback confirms itself months after the order, so you judge the autumn with the answers of the winter. Watch the gap between predicted regret and realized regret: that drift gives the first warning, well before any classical metric moves. Since that case I open projects differently: I write the cost matrix before choosing the model, then I have it signed by someone whose budget depends on it.
A success, once defined, still has to be protected. "When a measure becomes a target, it ceases to be a good measure": people credit that phrase to Goodhart, yet we owe it to the anthropologist Marilyn Strathern14, 15. Learning theorists have since turned it into a theorem: any indicator simpler than the objective it replaces can be gamed16. So stop hunting for the perfect indicator. Watch for a divergence.
I learned this the expensive way. On a real-time audio orchestrator, we had a speed threshold, the real-time factor or RTF. That ratio of computation time to signal duration had to stay under 1. The optimization treated it as a grade to minimize without end. It reached an RTF of 0.12, magnificent, while the transcription error rate rose by 300%. The system ran eight times faster than asked and made four times the errors. We repaired the constraint and never touched the model.
In 1974 Richard Feynman told the story of Pacific islanders who, after the war, rebuilt an airstrip from memory: the fires, the hut, the bamboo antennas, the man waving the signals. Everything is exact, the form is perfect and no plane lands17. He called that cargo cult science. Our field is very good at this game of forms: benchmarks that grade models, dashboards, A/B tests that compare two versions. All of these instruments exist for real and all of them also get made out of bamboo.
One test tells them apart, salutary in its brutality. Has this instrument ever changed a decision? And had the number come out differently, would it have changed it the other way? A dashboard nobody looks at is a stage set. Feynman stopped the analogy at a specific point and I stop it at the same one: these practitioners are sincere, neither frauds nor idlers. They lack a discipline toward their own minds, the one that debugs good surprises with the same energy as bad ones.
It costs little, this discipline. It pays handsomely, because intuition is wrong with great confidence. Take a 2025 randomized study of seasoned developers using code assistants. They believed they were 20% faster. They were 19% slower18. Nearly forty points separate the feeling from the stopwatch. The saying "what gets measured gets managed" is attributed to Peter Drucker. W. Edwards Deming called it a costly myth, because what matters most is often what you do not yet know how to measure19. Agree with both, in that order: first measure how bad you are, in the currency of your business, then distrust that measurement. That success still has to reach real hands.
Usable: the struggle
In 2020, GPT-3 already existed and the world was looking elsewhere. In November 2022, two years on, it took ChatGPT five days to change the public conversation. What moved? A short version of that story circulates, according to which only the interface changed. It is false: the brain changed too. It begins with a model in the GPT-3.5 series. Then comes training specific to dialogue: humans write answers, then the model learns from their preferences. That recipe had been published a few months earlier, in March 2022, under the name InstructGPT20, 21. Between "continue a text" and "follow an instruction" stands an entire training run. No stylesheet does that work.
Still, all that training aimed at one thing, letting anyone at all address that capability. So the door handle begins well before the text box. The story keeps its lesson all the same: a hundred capable models never found their handle. That is the struggle of the second U, the distance between beautiful theory and the hand that has to grasp it.
This struggle has its classic authors: Don Norman and his psychology of everyday things22, Steve Krug and his principle of least effort23. Artificial intelligence has its own, starting with the eighteen human-AI interaction guidelines Microsoft validated across twenty products24. The most useful notion with clients is calibrated trust: adjusting the user's trust to the system's actual capabilities instead of maximizing it25. Humans forgive machines poorly. A single visible error from an algorithm is enough to trigger abandonment where the same human error would be excused26. At the other extreme, people defer too much to the machine's number27. Between the two, design.
Let us size the stakes on one field study. Seven hundred and fifty-eight consultants at a major firm were randomly assigned to work with or without an AI assistant. Inside the domain where the model is competent, the assistant brought a quarter more speed and quality judged higher. Outside it, the assistant cost nineteen points of accuracy28. The authors speak of a jagged frontier: the machine's competence advances without the regularity of a tide, drawing headlands and coves invisible from the screen. The text box looks the same on both sides of that frontier. Users do not feel the moment they leave the zone where the machine excels for the one where it derails with aplomb. So the designer's first job is to make that frontier felt: show provenance, show uncertainty, invite verification when the ground turns soft.
The industrial designer Raymond Loewy, who drew locomotives as well as lipsticks, had a name for this: MAYA, Most Advanced Yet Acceptable. One notch too far and the human lets go. I turned it into an autonomy slider I set with my clients, in five notches: explain, suggest, draft, approve, act alone. One golden rule places the slider: the more an error costs, the less autonomy the artificial intelligence keeps.
The MAYA slider: five notches of autonomy, one golden rule to place it.
That setting is design in the full sense: governance sets the limits, design decides the human's place. One interface pushes toward decision, audit, correction. Another reduces the operator to approving without understanding. On the slides, both go by "human in the loop."
A validation that counts requires five conditions:
- The operator has time to read.
- The information is intelligible: a label, never an opaque identifier.
- The operator has the competence to judge.
- They have the right to refuse without penalty.
- They have a channel independent of what is being validated.
I have watched "ref. 58F2-C" flash up in front of operators ordered to approve at cadence: nobody validates an identifier, they endure it. Remove a single one of the five conditions and you get control theater. Multiplying confirmations on consequence-free actions trains the operator to click yes without reading. For Ivan Illich, a tool is convivial when it extends a person's autonomy, hostile when it enslaves the person to its upkeep29. The second U has no better definition, to my mind.
In the trade, this second U is called design: experience plus interface. The interface (UI, user interface) shows: the screen, the colors, the buttons, the chart, the sound. The experience (UX, user experience) puts you through something: the number of steps to the result, what you understand without explanation, what you dare to try without fear of breaking something, the trust that settles in or does not. The two get mixed up in meetings and the confusion costs real money. It happens that a superb interface dresses up a humiliating experience: three screens to validate one simple action. It happens that a well-thought-out experience dies of an illegible interface: the right path written in pale gray on white. The first U plays out in the numbers. The second plays out in the user's body, their eyes, their hands, their patience, their dignity. This split goes beyond personal taste: it can be measured, so it can be tooled.
My own tooling is a free-license software suite, sprezzature, published at sprezzature.ai30. It comes down to two words, build and audit. Every brick that produces ships with the auditor that judges it in a few milliseconds, before anything goes into service, whether a human or a machine wrote the code. That is how the demo films got their checker, which compares what the film claims to what the project contains. A real in-house bug gave birth to it: five films announced 127 chart types when the catalog held 167. Forty types lost in the pixels.
As long as design remains an opinion, it loses every arbitration against an indicator. Audited and blocking before release, it carries the weight of a failing test. The details are at sprezzature.ai. Yet the best-designed tool has proven nothing so far: the last gate is crossed over time.
Used: the freedom
Jacques Tati painted the portrait of the modern gadget: his houses purr with appliances that quietly manufacture more work than they save. Many artificial intelligence pilots end up the same way: everyone applauds the demonstration, then nobody logs in by week three. I have a test with nothing scientific about it and it never misses: three weeks from now, on an ordinary Thursday afternoon, will you still open this tool? A project that does not survive that plain Thursday is a screenshot.
Two fates for one launch: retention beats volume.
If you have ever published a video, you know a retention curve: YouTube shows creators, second after second, the share of the audience still watching. Mine counts people who come back, week after week. Alex Schultz, who ran growth at Facebook, drew from it in 2014 the simplest criterion in the trade. A retention curve that goes parallel to the time axis signals a product that holds. A curve that dives toward zero signals there is nothing to hold32. My Thursday-afternoon test is only the homemade version.
The third U has had a scholarly name since 1997. Parasuraman and Riley sort our relations with automation into four families, use, misuse, disuse, abuse33. Disuse covers the operators who abandon a reliable system after a few false alarms: silence settles in. Inside organizations the same abandonment is called shelfware, software bought at gold prices and never opened35.
Generative artificial intelligence has spread among individuals faster than any technology in history: a tenth of the world's adult population uses ChatGPT38. On the organization side, nearly all declare they use artificial intelligence, yet only 21% have redesigned a workflow40. And 78% of knowledge workers bring their own AI tools to the office, with no company approval41. Use overflows the official. I read that overflow the way a plumber reads a leak: an IT department that offers mediocre artificial intelligence manufactures its own shadow AI. That workaround accuses a product, not the employees.
So what do you do to be used? Three things, as prosaic as a maintenance contract: you train, you monitor, you maintain.
Training first. At a client that analyzes medical samples in the lab, success did not hang on the model. It hung on acculturating the whole chain, leadership, technicians, scientists, salespeople, with people in the loop labeling the examples the machine learns from. Your current team, trained up, is the cheapest artificial intelligence you will ever get. Training also has a dark side, which the work psychologist Lisanne Bainbridge named the ironies of automation as early as 1983: the human kept as the last resort is the one who has been detrained the most42. So I recommend everywhere a reserve of roughly 20% of work done without assistance, the way a pilot goes back to the simulator with every automation off.
Philosophers have an image for this erosion of competence: the ship of Theseus. Swap its planks out one by one until you wonder what is left of the original ship. Do the same to a team by handing an agent its work task after task. Then ask yourself, before it is too late: what remains of the competence that allowed anyone to supervise the agent? A technical executive told in public how he turned forty days of development over to an entirely artificial team, billions of compute tokens to show for it43. The account is enthusiastic and the question it leaves open is the right one: the day the agent is wrong, who on the team can still see it? The unassisted reserve is no purist's affectation: it keeps the human judge in working order.
Monitoring next. Overridden alerts are the rule: in medicine, 49 to 96% of prescription safety alerts are ignored44. A validator who has never refused has not validated. So I record three numbers: the refusal rate, the override rate, three-week retention. For me they are first-class indicators, ranked with the model's accuracy.
Maintenance finally. Research has established it firmly: in a real machine learning system, the model's code is a small box at the center of immense plumbing. Technical debt piles up there: everything put off for later makes each change more expensive45. CRISP-DM, the method a generation of data specialists trained on, relegates deployment to the sixth and last phase, barely described46. The engineers surveyed by research say it plainly: you have no idea how a model behaves in production before production47. A talk I often give took its title from that, the plumbing of artificial intelligence: the 90% nobody photographs48.
Two recent fables from the sector finish the demonstration. Zillow gave a price model the power to buy houses: predicting had become acting and the company wrote down 304 million dollars49. Klarna turned its customer support over to agents, praised itself in public50, then retraced its steps51. You could shrug and say "the model was bad." The real fault was allowing the model to act without review in a seat where errors are expensive. So the fix lies in the permission: the model can stay, but it loses the right to act unreviewed.
The small box and the plumbing, after Sculley et al. (2015): the model's code is the smallest piece of the system.
Something strange happens when these three jobs are done: artificial intelligence stops being announced. Even its name can drop out of the conversation. In 2023, Microsoft said "AI" twelve times in its earnings call, Google one hundred and forty-three times in two hours of conference (more than once a minute), Apple zero. Artificial intelligence that nobody names anymore, because it is simply part of the tool, is the finished form of the third U. If your initiative still needs a launch event and a reminder email, you have delivered a tool, not a use.
This is where freedom takes its full meaning. A sentence surprises people at the end of my deployment contracts: we organize my exit. I deliver an integration guide, I train, I provide support, then I leave. A successful project no longer needs me. I also teach for that reason, in an EMBA since 2022 and elsewhere: passing it on is the generalized form of leaving.
When agents join in
All of the above predates agents. They amplify all of it and abolish none.
An agent, in the sense the industry has given the word since 2024, is a large language model, of the kind behind ChatGPT, that directs its own process: it perceives, plans, acts through tools, loops52. Anthropic draws a useful line between the workflow, a predefined path that orchestrates a model, and the agent, which chooses its path53. The builder adds a counterintuitive piece of advice: start simple, add autonomy only if it pays.
One thing has changed in nature, though: the technical barrier has collapsed. One afternoon is now all an agent needs to write code that runs, stand up a mock-up, wire a database. That used to take a quarter. I treat a coding agent the way our elders treated the compiler. That program, which translates code into machine instructions, was a rise in abstraction: it displaced programmers' judgment without eliminating them. Yet building distinguishes nobody once everyone builds. The prototype becomes nearly free, so the sieve gains the value that technique loses. What remains is knowing what to build (useful), for which hand (usable), with what follow-through (used). The lower the technical barrier drops, the more the three Us become the profession.
Take useful: agents aggravate the original sin, coding before defining success. Gartner predicts the cancellation of more than 40% of agentic artificial intelligence projects by the end of 2027, citing rising costs, unclear business value and inadequate risk controls. It christens the disease agent washing54. At McKinsey the paradox comes with numbers: eight companies in ten deploy generative artificial intelligence, as many see no effect on the bottom line55. At Princeton, researchers find measurement itself broken: agent benchmarks optimized on accuracy alone, without cost or reproducibility, manufacture needlessly complex agents56.
Move on to usable. For the first time in sixty years, the very way we command a machine is changing: we specify intents instead of commanding actions57. Design now sets levels of supervised autonomy where it used to draw screens. The MAYA slider has never been so useful.
Get to used and you hit a wall called compounded reliability. The arithmetic fits on a bus ticket. Suppose an agent 95% reliable at each step, which is flattering, then give it a twenty-step task. Everything must succeed, so 0.95 multiplied by itself twenty times: barely more than one chance in three of reaching the end. At fifty steps, less than one chance in ten. So agents brilliant in demonstration collapse over distance.
Measurements confirmed it brutally at the outset. When OSWorld came out in 2024, a benchmark built from real office tasks, humans succeeded at 72% of them, the best agent at 12%58. Those two numbers are two years old and writing them in the present tense would be a lie. On July 28, 2025, the benchmark was taken back and cleaned, so scores from before and after do not line up59. They have climbed fast since. One piece of work clears the human bar in its February 2026 revision, 72.6% against 72.36%60. Check the protocol before you conclude anything. The agent runs several times on the same task and a judge picks the best attempt. That record tells you what an agent reaches when you pay for several tries. Getting it right the first time is another promise, at another price.
Facing simulated users under business rules, the best models fail more than half the time. Hand them the same task eight times and the share of tasks completed every time collapses further, because the agent does not always miss the same step61. For customer service, succeeding often falls short: succeeding every time is what makes trust. The trajectory is real: the length of the tasks an agent can complete doubles roughly every seven months62. A trajectory is still no guarantee: it is a slope, to be watched like every divergence.
And there is the bitter lesson. Richard Sutton observed in 1,100 words that over sixty years, the general methods that convert compute into performance end up beating the trade rules patiently written by hand by experts63. Chess, go, speech, vision: four times the same story. Economics explains it, with two clocks for a single budget: your project counts in quarters, the cost of compute falls on a decade-long curve. Hand-written rules are frozen capital that ages, the general method is a subscription to a price drop. One purchasing question settles it: what does your system do with ten times more compute? If the answer is "it goes faster," you are looking at a rule base. If the answer is "it gets better," you are looking at a system that learns.
The lesson has a limit, though: all the winners of this story were hybrids. Deep Blue, the IBM computer that beat the world chess champion in 1997, shipped with an evaluation function of eight thousand hand-tuned features on top of its computing power. Sutton's critics rightly point out that human ingenuity does not disappear. It relocates64. And it relocates toward our three Us: if compute takes the how, the human keeps the what, the for whom and the what next. The practitioner comes out of this with judgment worth more.
Two clocks, one budget: a project's horizon against the compute curve.
How do you work with agents without handing them the helm? Two disciplines help. The first is a public method, BMAD, for Breakthrough Method of Agile AI-Driven Development. It simulates a complete software team with specialized agents that pass work along the way a human analyst, product manager, architect, developer and tester would65. Its creator sees it as the opposite of vibe coding66. Its virtue is an inversion: the planning phase forces the brief, the product requirements and the architecture to be written before the first line of code. A human validates each artifact.
In our terms, BMAD institutionalizes the first U. It writes the second into the artifacts, since user experience specifications are a deliverable there instead of a courtesy. It then paces the third: each development task closes with a review. The reflexes of agile development are back on duty, as guardrails for machines in a hurry.
The second discipline, a handcrafted one, I have described in detail elsewhere: Ralph loops67. Act, render, judge, decide, in a loop, with a budget of attempts declared in advance and a protected judge, because an agent that optimizes its judge ends up corrupting it. Goodhart takes no vacations. Feynman's discipline, set to machine rhythm: never let the one who produced the result tell you what it says. The figures in this article went through that loop, defects logged then fixed at the source. The method applies to itself.
The Ralph loop: four steps, one budget, a judge that cannot be bought.
The Monday-morning compass
A framework that never reaches Monday morning is worth nothing. So on every project I ask these questions, in this order. None requires knowing how to program and each has already saved or killed a project before my eyes.
For the useful. What is the unit? How much is each type of error worth in euros, in minutes, in care? Which line of the year's accounts must move, by how much, by when? Then the hardest one, which I owe to Feynman: if the number had come out the other way, would we have decided differently? If not, your instrument is bamboo.
For the usable. Who will open it on a Tuesday, tired, in a hurry, with no fresh training? How far does the machine's competence go and what on the screen tells the user so? On which notch of the slider does autonomy rest, who has the right to raise it, who has the duty to lower it when the cost of error climbs?
For the used. Who gets trained, on what, and how much of the work stays unassisted? Who tracks validators' refusal rates and three-week retention? On which budget line will maintenance get paid next year and by whom? Last comes the clause that forces everyone into seriousness at the signature: how do we organize my exit?
Dies of freedom
That leaves Gide's last verb: art dies of freedom. The work without constraint dissolves into ease, he says. For the work, I grant him that. On my projects his sentence has come true twice, in two opposite directions. They need keeping apart, since one describes a failure and the other describes the goal.
First direction: the freedom that kills. Free an artificial intelligence system of its constraints and you doom it. Remove the unit: it will optimize an empty number, like my audio orchestrator become eight times faster than necessary while its transcriptions degraded. Remove the independent judge: it will end up corrupting its own evaluation. Goodhart promised as much and the experience of agents confirms it weekly. Remove human review in a seat where errors are expensive: Zillow, as we saw, let its model buy houses in full freedom and the venture ended on a nine-figure write-down. The kite string, again: cut it to free the object and it falls. An unconstrained agent is an accident in the making. The three Us are the strings we leave it: a success defined before it, a place drawn for the human, a watch that does not fall asleep.
Second direction: the freedom that crowns. An artificial intelligence system is born of constraint: a real problem, a unit, a cost of error, a number to move. It lives on struggle: against the gap between theory and the field, for an interface that respects the hand and the mind of the person working. And it is fulfilled in freedom. Not its own, ours. The teams' first: a convivial tool, said Illich, extends a person's autonomy instead of enslaving them to its upkeep. A used system makes people freer in their own craft, faster toward what matters, relieved of what does not. The practitioner's next: putting away one's tools, passing on what one knows, leaving to begin a new constraint elsewhere. What dies then is not the project. It is the need anyone had of me.
The two directions hold together and their paradox pleases me: for freedom to crown the humans, it has to be denied to the machine. Give the system all its freedom and you take it from its users, condemned to watch, correct, repair. Gide was obviously not thinking of agents in 1904 and the transposition stops there: a stage play does not refund a house bought too dear. Yet the mechanism he described, the demand that elevates and the ease that dissolves, I find it in every steering committee.
Useful, usable, used. When the work is done well, the system runs on an ordinary Thursday afternoon and nobody thinks about it. That is all I wish it.
Bibliography
Each title below served a specific section, which its note names. None is cited secondhand.
- Gide, A. (1911). De l'évolution du théâtre. In Nouveaux Prétextes : réflexions sur quelques points de littérature et de morale, Mercure de France The 1904 Brussels lecture, "De l'évolution du théâtre," source of the sentence, the kite, Michelangelo and Aeschylus.
- Gide, A. (1904). Saül ; Le Roi Candaule. Société du Mercure de France The lecturer's two wounded plays, Saül and Le Roi Candaule: Gide spoke from experience of failure.
- Léonard de Vinci (1881). Les manuscrits de Léonard de Vinci : le manuscrit A de la Bibliothèque de l'Institut. A. Quantin Da Vinci's manuscript A: force "is born of violence and dies of freedom," the matrix of Gide's formula.
- Davis, F. D. (1989). Perceived Usefulness, Perceived Ease of Use, and User Acceptance of Information Technology. MIS Quarterly, 13(3), pp. 319-340 The technology acceptance model: perceived usefulness outweighs ease of use.
- Nielsen, J. (1993). Usability Engineering. Academic Press The founding handbook: utility multiplied by usability.
- International Organization for Standardization (2018). ISO 9241-11:2018 : Ergonomics of human-system interaction : Part 11: Usability: Definitions and concepts. Geneva: ISO, 2e édition The standard that defines usability in black and white: effectiveness, efficiency, satisfaction, in context.
- Tricot, A., Plégat-Soutjis, F., Camps, J. et al. (2003). Utilité, utilisabilité, acceptabilité : interpréter les relations entre trois dimensions de l'évaluation des EIAH. In Environnements Informatiques pour l'Apprentissage Humain (EIAH 2003) The French-speaking triptych: utility, usability, acceptability.
- Interaction Design Foundation (2016). Useful, Usable, and Used: Why They Matter to Designers. IxDF Literature, 17 mars 2016 English-speaking designers and their "useful, usable, used."
- Kothari, A., Sibbald, S. L., McCutcheon, C. et al. (2025). How does integrated knowledge translation work? A realist review. Health Research Policy and Systems, 23(102). doi.org Health research demands results that are "useful, usable and used."
- Gartner, I. (2018). Gartner Says Nearly Half of CIOs Are Planning to Deploy Artificial Intelligence. Press release, Stamford, CT, 13 février 2018 The 2018 press release behind the "85%": it actually predicted erroneous outcomes for 2022, never reevaluated.
- VB Staff (2019). Why Do 87% of Data Science Projects Never Make It into Production?. VentureBeat, 19 juillet 2019 The sponsored article behind the "87%," with no primary source.
- Challapally, A., Pease, C., Raskar, R. and Chari, P. (2025). The GenAI Divide: State of AI in Business 2025. MIT NANDA (Project NANDA, Massachusetts Institute of Technology) The MIT report of the "95%," "directionally accurate" by its own admission.
- Ryseff, J., De Bruhl, B. F. and Newberry, S. J. (2024). The Root Causes of Failure for Artificial Intelligence Projects and How They Can Succeed: Avoiding the Anti-Patterns of AI. RAND Corporation The RAND interview study. It borrows the figure of more than 80% failures "by some estimates" and finds, on its own, that the leading cause is a misunderstanding of the problem to solve.
- Strathern, M. (1997). `Improving ratings': audit in the British University system. European Review, 5(3), pp. 305-321 The exact phrase "when a measure becomes a target," which is hers and not Goodhart's.
- Goodhart, C. A. E. (1984). Problems of Monetary Management: The UK Experience. In Monetary Theory and Practice: The UK Experience, Macmillan Education UK. doi.org Goodhart's text, whose name the law carries.
- Skalse, J., Howe, N. H. R., Krasheninnikov, D. and Krueger, D. (2022). Defining and Characterizing Reward Hacking. In Advances in Neural Information Processing Systems 35 (NeurIPS 2022) The theorem: any indicator simpler than the objective it replaces can be gamed.
- Feynman, R. P. (1974). Cargo Cult Science. Engineering and Science, 37(7), pp. 10-13 The cargo cult science address, and the discipline toward one's own mind.
- Becker, J., Rush, N., Barnes, E. and Rein, D. (2025). Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity. arxiv.org The randomized study: developers who believed they were 20% faster and were 19% slower.
- Deming, W. E. (n.d.). The New Economics for Industry, Government, Education. MIT Center for Advanced Engineering Study Deming against the measurement saying: what matters most is often what you cannot yet measure.
- OpenAI (2022). Introducing ChatGPT. openai.com The ChatGPT announcement, where OpenAI itself names the starting model, the GPT-3.5 series, and the dialogue training: the source that forbids reducing the 2022 jump to an interface.
- Ouyang, L., Wu, J., Jiang, X. and others (2022). Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems. arxiv.org InstructGPT, the human-preference training recipe ChatGPT reuses: what changed inside the brain, not only on the screen.
- Norman, D. (2013). The Design of Everyday Things: Revised and Expanded Edition. Basic Books The psychology of everyday things: the door handle that tells you how to grab it.
- Krug, S. (2014). Don't Make Me Think, Revisited: A Common Sense Approach to Web Usability. New Riders The principle of least effort: don't make me think.
- Amershi, S., Weld, D., Vorvoreanu, M. et al. (2019). Guidelines for Human-AI Interaction. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI '19), ACM The eighteen human-AI interaction guidelines, validated across twenty products.
- Lee, J. D. and See, K. A. (2004). Trust in Automation: Designing for Appropriate Reliance. Human Factors, 46(1), pp. 50-80 Calibrated trust: adjusting the user's trust to the system's actual capabilities.
- Dietvorst, B. J., Simmons, J. P. and Massey, C. (2015). Algorithm Aversion: People Erroneously Avoid Algorithms After Seeing Them Err. Journal of Experimental Psychology: General, 144(1), pp. 114-126 Algorithm aversion: a single visible machine error is enough to trigger abandonment.
- Logg, J. M., Minson, J. A. and Moore, D. A. (2019). Algorithm Appreciation: People Prefer Algorithmic to Human Judgment. Organizational Behavior and Human Decision Processes, 151, pp. 90-103 The opposite effect: excessive deference to the machine's number.
- Dell'Acqua, F., McFowland III, E., Mollick, E. et al. (2026). Navigating the Jagged Technological Frontier: Field Experimental Evidence of the Effects of Artificial Intelligence on Knowledge Worker Productivity and Quality. Organization Science. doi.org The 758 consultants and the jagged frontier: faster inside, wrong outside.
- Illich, I. (1973). Tools for Conviviality. Harper & Row The convivial tool, which extends a person's autonomy instead of enslaving them to its upkeep.
- Harchaoui, W. (2026). Sprezzature : create and audit figures, visualizations for app and web content. sprezzature.ai} The suite itself: build and audit, all local, documentation and gallery.
- Castiglione, B. (1991). Le Livre du courtisan. Flammarion The Courtier, where sprezzatura is the art of hiding the art.
- Schultz, A. (2014). Lecture 6: Growth. Stanford CS183B “How to Start a Startup”, Y Combinator. youtube.com Schultz's lecture: the retention curve that flattens, the simplest criterion in the trade.
- Parasuraman, R. and Riley, V. (1997). Humans and Automation: Use, Misuse, Disuse, Abuse. Human Factors, 39(2), pp. 230-253 The taxonomy use, misuse, disuse, abuse: the scholarly name of the third U.
- Parasuraman, R. and Manzey, D. H. (2010). Complacency and Bias in Human Use of Automation: An Attentional Integration. Human Factors, 52(3), pp. 381-410 Complacency: trusting the machine without checking anymore.
- Markus, M. L. and Keil, M. (1994). If We Build It, They Will Come: Designing Information Systems That People Want to Use. Sloan Management Review, 35(4) Shelfware, bought at gold prices and never used.
- Moore, G. A. (2014). Crossing the Chasm: Marketing and Selling Disruptive Products to Mainstream Customers. HarperBusiness The chasm between demo enthusiasts and the pragmatic majority.
- Selwyn, N. (2003). Apart from Technology: Understanding People's Non-Use of Information and Communication Technologies in Everyday Life. Technology in Society, 25(1), pp. 99-116 Non-use as an often rational choice, not a deficit to correct.
- Chatterji, A., Cunningham, T., Deming, D. J. et al. (2025). How People Use ChatGPT. National Bureau of Economic Research A tenth of the world's adult population uses ChatGPT.
- Brynjolfsson, E., Li, D. and Raymond, L. (2025). Generative AI at Work. The Quarterly Journal of Economics, 140(2), pp. 889-942. doi.org The 15% productivity measured among support agents, novices above all.
- McKinsey & Company (2025). The State of AI: How Organizations Are Rewiring to Capture Value. McKinsey & Company, QuantumBlack Nearly all organizations declare they use AI; 21% have redesigned a workflow.
- Microsoft and LinkedIn (2024). AI at Work Is Here. Now Comes the Hard Part: 2024 Work Trend Index Annual Report. Microsoft Corporation The 78% of knowledge workers who bring their own AI tools to the office.
- Bainbridge, L. (1983). Ironies of Automation. Automatica, 19(6), pp. 775-779 The ironies of automation: the last-resort human is the most detrained one.
- Bonnin, L. (2026). J'ai viré mon équipe de dev pour une équipe 100\,% IA pendant 40 jours et 2,3 milliards de tokens The tale of forty days of development handed to an entirely artificial team.
- van der Sijs, H., Aarts, J., Vulto, A. and Berg, M. (2006). Overriding of Drug Safety Alerts in Computerized Physician Order Entry. Journal of the American Medical Informatics Association, 13(2), pp. 138-147 The 49 to 96% of prescription alerts ignored: the measurement of overriding.
- Sculley, D., Holt, G., Golovin, D. et al. (2015). Hidden Technical Debt in Machine Learning Systems. In Advances in Neural Information Processing Systems 28 (NIPS 2015) The small box and the plumbing: the hidden technical debt of machine learning systems.
- Chapman, P., Clinton, J., Kerber, R. et al. (n.d.). CRISP-DM 1.0: Step-by-Step Data Mining Guide. The CRISP-DM Consortium (NCR, SPSS, DaimlerChrysler) CRISP-DM, which relegates deployment to the sixth and last phase, barely described.
- Shankar, S., Garcia, R., Hellerstein, J. M. and Parameswaran, A. G. (2024). ``We Have No Idea How Models Will Behave in Production until Production'': How Engineers Operationalize Machine Learning. Proceedings of the ACM on Human-Computer Interaction, 8(CSCW1), pp. 206:1-206:34 The engineers surveyed say it: you do not know a model before production.
- Harchaoui, W. (2026). Conférences en intelligence artificielle. deraison.ai My talks, including the plumbing of artificial intelligence.
- Zillow Group (2021). Zillow Group Reports Third-Quarter 2021 Financial Results & Shares Plan to Wind Down Zillow Offers Operations Zillow: predicting had become acting, 304 million dollars written down.
- Klarna (2024). Klarna AI assistant handles two-thirds of customer service chats in its first month Klarna publicly congratulates itself on its support agents.
- Ekblom, J. (2025). Klarna Turns From AI to Real-Person Customer Service. bloomberg.com Then backtracks.
- Wang, L., Ma, C., Feng, X. and others (2024). A Survey on Large Language Model Based Autonomous Agents. Frontiers of Computer Science. doi.org The state of the art on agents: perceive, plan, act through tools, loop.
- Anthropic (2024). Building Effective Agents. anthropic.com Workflow versus agent and the counterintuitive advice to start simple.
- Gartner, I. (2025). Gartner Predicts Over 40% of Agentic AI Projects Will Be Canceled by End of 2027. Communiqué de presse, 25 juin 2025 The predicted 40% of agentic projects canceled by the end of 2027 and the disease's name: agent washing.
- McKinsey & Company (QuantumBlack) (2025). Seizing the Agentic AI Advantage The paradox: eight companies in ten deploy, as many see no effect on the bottom line.
- Kapoor, S., Stroebl, B., Siegel, Z. S. et al. (2024). AI Agents That Matter. arxiv.org The broken agent benchmarks: accuracy alone, without cost or reproducibility.
- Nielsen, J. (2023). AI: First New UI Paradigm in 60 Years. Nielsen Norman Group, 18 juin 2023 The first interface paradigm shift in sixty years: specifying intents.
- Xie, T., Zhang, D., Chen, J. and others (2024). OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments. arxiv.org OSWorld: 72% human success on real office tasks, 12% for the best agent.
- OSWorld Team (2025). OSWorld-Verified. osworld-v1.xlang.ai The benchmark's own site, dating its corrected edition to July 28, 2025: why scores from before and after do not line up.
- Gonzalez-Pumariega, G., Tu, V., Lee, C. and others (2025). The Unreasonable Effectiveness of Scaling Agents for Computer Use. arxiv.org The work that clears the human bar on OSWorld in its February 2026 revision, by running the agent several times and letting a judge pick: the record that forces you to read its protocol.
- Yao, S., Shinn, N., Razavi, P. and Narasimhan, K. (2024). $\tau$-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains. arxiv.org Facing simulated users under business rules: failure more than half the time, and not always on the same step.
- Kwa, T., West, B., Becker, J. and others (2025). Measuring AI Ability to Complete Long Software Tasks. arxiv.org The horizon of succeeded tasks doubles roughly every seven months: a slope, not a guarantee.
- Sutton, R. S. (2019). The Bitter Lesson. incompleteideas.net, 13 mars 2019 The bitter lesson: over sixty years, compute ends up beating hand-written rules.
- Brooks, R. (2019). A Better Lesson. rodneybrooks.com, 19 mars 2019 Brooks's reply: human ingenuity does not disappear, it moves.
- Madison, B. (2025). BMAD-METHOD: Breakthrough Method of Agile AI-Driven Development. github.com The BMAD method: a complete software team simulated by specialized agents.
- Madison, B. and Prasetya, H. (2026). Stop Vibe Coding: Spec-Driven Development with The BMad Method. Tech Lead Journal, épisode 255, 20 avril 2026 Its creator presents it as the antithesis of vibe coding.
- Harchaoui, W. (2026). Tant que ce n'est pas bon. deraison.ai This article's companion piece: the loop, its budget declared in advance and its protected judge.