#### Agent Identity: How to Tell If a Bot Is Who It Claims to Be Picture a request in your logs at three in the morning. The user-agent says Googlebot, the IP belongs to some residential ISP, and in forty minutes about 4,300 pages were read without a single pause. Anyone running a site with some traffic has seen a version of this. The first instinct is to block the IP, and it’s the wrong one: tomorrow another will show up. The problem starts before the IP. #### <br>The user-agent is a business card written in pencil It’s a line of text the client sends of its own accord. Nothing in HTTP forces anyone to tell the truth there, and any twenty-line script can write “Googlebot/2.1” or “Bingbot”. That’s why any serious reasoning about bots starts by accepting that the user-agent declares an intention and never proves an identity. Declaring has value, mind you. An honest crawler says who it is so you can apply rules in robots.txt. But robots.txt is a courtesy request, and whoever doesn’t want to respect it won’t. #### <br>The old-school check The big search engines solved this long ago with DNS. You take the request’s IP, do a reverse lookup and get a hostname. For Google, that name must end in googlebot.com or google.com. Then you do a forward lookup on that name and confirm it returns the same IP. If the loop closes, the request is legitimate. Bing has an equivalent process, and both publish IP range lists as JSON files. It works well and has two practical flaws. Each operator has its own list and its own format, so anyone verifying a dozen agents ends up with a dozen tedious maintenance scripts. And the lists go stale. I’ve seen sites block genuine crawlers because an IP range changed and nobody updated the file (it happened to me once, and I spent two days convinced the problem was the cache). The underlying idea is sound. Identity rests on something the impostor doesn’t control, which here is network infrastructure. #### <br>Signatures: the new part The newer approach puts the IP in the background and cryptography at the center. The agent signs each request with a private key, and the server verifies the signature with the matching public key, published at a known address. The technical basis is RFC 9421, HTTP message signatures, published in 2024. On top of it came the proposal called Web Bot Auth, which has been discussed at the IETF and which Cloudflare adopted in its verified bots program. An extra header says where to find the agent’s key. The practical advantage is big. An AI agent running across different clouds, with rotating IPs, becomes recognizable without an IP list. The signature proves the request came from whoever holds the key, and that’s enough. OpenAI was among the first to sign its agents’ requests this way, but adoption changes fast, so check each operator’s documentation before building rules on top of this. #### <br>Identity is not intent This is where the conversation gets messier. A valid signature tells you who made the request. It doesn’t tell you whether the request is reasonable, or whether the operator respects your limits. The case that illustrates this best happened in August 2025, when Cloudflare accused Perplexity of getting around blocks with crawlers that didn’t identify themselves as such. Perplexity disputed that reading. I’m not interested in deciding who was right. What interests me is what the episode shows: blocking by name fails against those who don’t use the name, and perfect identification does nothing to stop someone reading 4,000 pages a minute. And there’s a third type that complicates everything. An agent browsing on behalf of a person, booking a flight or comparing prices, is automation and is also someone with a real need. The “good bot, bad bot” split can’t handle this case. Imperva’s 2025 report put more than half of 2024 web traffic as automated, with around 37% malicious bots. With half the traffic being machines, treating everything non-human as suspect has stopped making sense. To me it seems more useful to ask four things, in this order. Who is it, and can I prove it? Who sent it? What does it want from me? And what am I willing to give it, and at what price? People skip that last question, and it’s where rate limits, gated access and the pay-per-crawl models Cloudflare launched in 2025 start to make sense. I’m not sure universal signatures will ever become the norm. But the user-agent as proof died a long time ago, and it’s still the first filter for almost everyone.