Sorry the blog has been so quiet. I didn’t realise that even my published articles weren’t actually being published. I finally took a look after getting an email from Jetpack that my blog was “intermittently offline”, and when I logged in to the server that runs this blog (along with a few other services), I poked around and everything seemed to be running. When I logged off, I was greeted with a strange and somewhat concerning message1:
zsh: failed to write history file /home/spots/.zsh_history: No space left on deviceOh. Ohh…
I checked /var/log and alas, the horror: 148 GiB of AI bot crawler logs accumulated over the past week. They found code.atwilcox.tech! We’re all doomed. I removed those logs, reloaded Apache, and now everything’s working again.
I suppose I’ll need to change the log rotation on GitLab to hourly now, and probably only retain for a day. Thankfully, mercifully, the bots don’t seem to have found any of the other sites on this server. The whole year’s traffic for this blog is under 300 MiB, and I think that’s still a lot of bot traffic.
I really don’t want to use that cheesy anti-Chrome hack that I used to protect code.foxkit.us, but if this keeps up, I will do it in the interests of self-preservation. I don’t want to punish well-crafted bots2, and I don’t want to punish people who know what they’re doing3. But I suppose neither of those will have a Chrome user agent, now, will they4…
It feels like surely we must be hitting the end of “burn the Earth to train these models”, right? The good models have terabytes of training material now. The startup models shouldn’t be crawling like this because it’s going to waste their resources, and I’ve heard VC isn’t as easy to come by any more. So who, in September 2026, is still doing training this way? I don’t understand…
- Message changed slightly because I don’t want to leak the actual account name I use on the prod server, in the interests of defence-in-depth. ↩︎
- Web Archive, CCBot, search engine indexers, and assistive technologies, to name a few. ↩︎
- I’m not even averse to using an AI/LLM agent to crawl through the repository if it actually yields a meaningful result. Need Claude to look through the Wii U package repository we host to cross-reference something? Probably wasteful, but go for it. But these dumb bots can’t even figure out that they’re in permalink hell, getting every SHA of every file… ↩︎
- And yes, I am implying that people who know what they’re doing don’t use Chrome. Most of my friends are either Safari or Firefox users. The ones who are still Chrome users just haven’t gotten the Spots Talk About Chrome™ yet. ↩︎