1793614433

Self-Healing Code Is Just Code That Learned to Hide Its Errors


The first time I saw this work for real, it was by accident. I had a Node worker that called an external API through a keep-alive agent and died with ECONNRESET about three times a week, almost always between 3 and 4 in the morning. Lazy as I am, I put Restart=on-failure and RestartSec=5 in the systemd unit and forgot about it. For four months the service healed itself. Nobody noticed, me included, until the day I went to read journalctl for some other reason and found 212 restarts in a single month. People call this self-healing code. I call it something else, and we'll get there. It helps to separate two families that marketing mixes up on purpose. The first is the old one: supervisors, retries with backoff, circuit breakers, Kubernetes liveness probes killing pods that stopped responding. It comes from the Erlang school, from Armstrong's "let it crash", and it has an honesty I find admirable, because it doesn't pretend to fix anything. It returns the system to a known good state, and that's it. It fixes the state, not the code. The second family is the recent one, the one that lives in investor decks: an AI agent that reads the stack trace, opens the file, writes a patch and submits a pull request, all wired to Sentry through a webhook. This one does pretend. And it pretends well. Even the old family bites when you trust it too much. I once watched a liveness probe with timeoutSeconds: 1 kill perfectly healthy pods during a traffic spike, which pushed the load onto the survivors, which were also killed, and so on for 23 minutes of dominoes. The system was trying to heal itself and what it did was kill itself, methodically. I tried the second family in the spring, on a personal project. Sentry webhook, a script of about 140 lines, a model reading the error and the file, an automatic branch, a PR. The first week brought 11 pull requests and I merged seven. I felt that strange thing of having an intern who never sleeps, and I'll admit that for a few days I thought the rest of the profession was settled. Then came the TypeError: Cannot read properties of undefined (reading 'map'). The agent did what it was asked, which was to make the error go away. It put a `?? []` in the right place, the test passed, Sentry went quiet. Except the undefined came from a third-party API returning null because the token had expired. Result: empty lists, not a single exception, for nine hours. The error graphs had never looked so good and the product had never been so broken. Silencing is the cheapest way to fix. This seems to me the core of everything, and it isn't a model bug, it's a property of the design. A system that heals itself optimizes what it can measure, and what it can measure is the absence of the symptom. Sentry is a smoke detector. Taking the batteries out solves the beeping, and these agents do it with touching ease. It's the second-order effects that keep me up at night. First, the telemetry loses meaning: if half the errors are absorbed by invented default values, the graph stops telling the truth about the product, and you start deciding based on an instrument calibrated to please you. Then the team (or the lone guy, in my case) stops reading errors, because the bot handles it. The muscle atrophies. When the bot fails, and it will, nobody knows the code well enough to understand what it has been patching over the last six weeks. And there's review fatigue, the sneakiest of all: a four-line diff is too small for anyone to read seriously. You approve it in forty seconds. Eleven four-line PRs are more dangerous than one with four hundred. I remember an empty try/catch I left in a PHP 5.6 shop in 2014, "temporary", for an error that only happened on Fridays. It was still there six years later when I left the company. I never found out why it was Fridays. Anyway. It would be unfair, or at least incomplete, to say it's worthless. It's worth something where an oracle exists. If the success criterion is something other than the absence of the error, a decent test suite, a demanding type checker, then letting the machine work is reasonable. Patch-level dependency updates, mechanical deprecations, codemods, formatting, migrations with verification. About 37% of last quarter's commits in my main repository were of this kind, and I wouldn't go back. So my position is more or less this: what works is the boring part, and the boring part rarely makes it onto the slide. There's a cost almost nobody mentions, and it's literally money. One night my loop failed the test, retried, failed, retried, each iteration with more context stuck to the previous one, and I woke up to a 41 dollar bill for a fix that in the end didn't even compile. Retry without a ceiling is an open tap. The same goes for classic supervisors, by the way: a process with a memory leak that restarts every six hours is a process whose leak nobody will fix for two years, because the symptom never bothers anyone. And then there's the structural flaw, the one that seems deepest to me and gets talked about least. For a system to heal, it has to know what healthy means, and defining that is exactly the hard work of engineering. How many `/health` endpoints return 200 just because Express is alive, while the database has been read-only for twenty minutes? Liveness is not readiness, and no probe knows the difference between "the process responds" and "the product does what it promised". An agent asked to fix this inherits the same blindness, with more confidence. There's something I still haven't settled with myself. The idea of self-correcting code assumes the error is a defect in the code. Often it's a defect in the expectation: the token expired, the API contract changed, the user did something nobody predicted. You don't patch that. But I'm not sure it's always like that either, and there are probably whole classes of errors where the machine is right and I'm just a grumpy old man. The doubt stays. If someone insists on plugging an agent into production, here's how I'd do it, with little patience for debate. The agent proposes, never merges. It has to write a failing test first and only then the patch, and the test must fail before and pass after, otherwise the PR doesn't even open. Any new `catch` or new `?.` without an associated test is rejected outright. And the original Sentry alert stays alive for seven days after the "fix", precisely to catch the manufactured silence. It sounds like bureaucracy. It's the price of not waking up at 3:12 (because, by the way, it's always 3:12 when things fall over, and I never figured out whether the cause is the backup cron or a law of nature nobody wrote down). As for my worker. I finally found the cause on a Sunday afternoon: the remote server closed idle sockets before my agent considered them dead, and I was reusing a connection that was already buried. I fixed it with one line, keepAlive: false on the client. Four months of automatic healing on top of one line.

(0) Comments

Welcome to Chat-to.dev, a space for both novice and experienced programmers to chat about programming and share code in their posts.

About | Privacy | Donate
[2026 © Chat-to.dev]