This should have been a great day at work since I had all my sleep and was fully rested and Mum was handling my grandmother pretty well.
It was not a great day at work.
We all spent half the day fighting fires from the LLM bot attacks. Again. And trying to implement more mitigations and band-aids to weather the storm. Again.
Someone shared an article about “the inevitability of LLM crawlers” in a shared Library space, and I’m going to print my response here verbatim. I think it’s important to say this to as wide an audience as will listen to me.
The inevitability tone of the end of this article makes me roll my eyes so hard it could have given me eye strain. I don’t believe that this is either how any of this should behave, nor do I think it will be how any of this will behave in the future. Massive parallel queries like that, “millions” to answer a single user’s question (hyperbole more than likely, but still), are a huge waste of bandwidth and processing time. Right now, most LLM companies are still highly subsidised, but that won’t be the case forever. Any org that can make it to the other side [of the LLM bubble], whatever that may be, is going to regret wasting precious resources building the sort of infrastructure described in the article, IMO.
I explicitly reject the concept that regular botnet attacks on infrastructure from AI/LLM training or “agentic search” or whatever is something that will be with us very long. It is simply not sustainable and evolutionary pressures will end it. The market will force the subsidies to end eventually, and when it does, so do the millions of useless requests per day.
This is not the new normal to which we should all be adjusting. This is the war to endure before calm finally finds us, weary and shipwrecked but somehow alive on the shore, clinging to the sand that no longer thinks…