This is normally where I’d put that notice about me being on my tech soapbox again. But really, this article is worth reading no matter what you do, or what your interests are.
I had a friend come to me in a bit of a depressive spiral last week. She had yet another case of AI Overload, that is, “everything is AI/LLM now!” and felt like the entire tech industry is going to crash down around and on top of her. We had a deep conversation about it and both agreed it would be a good thing for me to turn into an article.
I’m going to gloss over the issues of copyright here. Anything that an AI or LLM generates, in the United States at least, cannot be copyrighted. That means anything you vibe code could be legally public domain (but who can tell with the courts these days). Not entirely great if you want to make a product out of it with most business models. It also means any art created is public domain – imagine if Disney used AI to create a film. They couldn’t copyright it. We would be able to do whatever we want with it, and they’d have no legal recourse. Not something you can see the House of Mouse doing, is it?
I’m also going to gloss over the fact that Anthropic was bragging about getting “37% correct” on a software engineering test with the Fable 5 model. At least, they were, before Fable 5 was pulled because the White House hates Anthropic for being anti-exploding-drone. That means they were bragging about being 63% incorrect.
The point is: that 37% they can automate is a lot of boilerplate. In the proper hands, with good engineering and a strong architecture and well-documented requirements, that 37% can genuinely reduce the time it takes to create reliable, secure, functional software.
The flip side is that if you give LLMs the other 63%, you’re going to have a nightmare, hellscape, rotten mess. And that 37% correctness was touted as being twice as accurate at GPT-5o. That’s not a great look for any of the present frontier models.
Automating the boring away != automating the job away
Let’s approach this by looking at some of the commonly hailed advances of LLMs.
Test case generation
I can easily tell one of these models “I wrote a bunch of tests for my Patron model, please write similar tests for my User model”, and they will generally do a decent job. At the very least, it will generate a useful scaffold I can refine later.
This saves a lot of time, but it isn’t really engineering. It’s rote copy and paste with a bit of pattern replace. It’s saving keystrokes, which is something we should be happy about, but it isn’t saving any engineering work, which is why “replace jobs with LLM” seems so ludicrous to me.
This is basically a supercharged, fancy version of rails generate. It makes scaffolding for you that can save a lot of time, entirely because there’s nothing unique or special about it. There’s no art or feeling or problem solving involved.
Generic Business CMS #1,539,153,859,159
Note: CMS, in this context, means “content management system” – think software like WordPress, Drupal, SharePoint, or such, but custom.
Every business eventually invents its own content management system. I don’t think I’ve ever worked at a place that didn’t. Even the startups had their own custom MediaWiki or WordPress plugins, and the bigger institutions had their own custom systems outright. This is not an interesting problem to solve; everyone has done it to death, and everyone pretty much hates it.
The fact that an LLM can write a somewhat-convincing CMS is a testament to 1. how many there are, 2. how easy they are to write, and 3. how fundamentally unoriginal they are. Writing a CMS is not something you do to satisfy the need to create things as an engineer or programmer. Writing a CMS is something you to satisfy a business requirement.
It would be similar if I asked an LLM to draw me a picture of “a dog”. It can probably do that, because there are billions of dogs in the world, and probably trillions of photos of dogs it has been trained on. It’s going to perform a lot worse at drawing me a picture of “a juvenile Dalmatian with a specific fur pattern standing atop an adult Dalmatian, looking at each other sweetly”, because there are very few (if any) pictures of that it has been trained on, and that is an actually original idea.
In the 1990s, which I am unfortunately old enough to not only remember but participated in, we had tools like Delphi and Visual Basic. The promise was that you could drag-and-drop UI elements like buttons and text boxes and such onto a canvas (normally called a “Form”), then write a very small amount of code to connect those controls to a database. In some cases, and with some revisions of the software (thinking of Visual Basic 5/6 with ADO, or Visual FoxPro), you may not actually write any lines of code! LLMs are basically that, but with the ability to do it in JavaScript or Go.
Easy Things Aren’t Always Easy; Hard Things Are Always Hard
The problem is not that LLMs can magically take unoriginal ideas and output code that reasonably matches the requirements it has been fed. And let me clear: it cannot always do that, but I concede it is capable sometimes.
The problem is that CEOs and upper managers of companies see this, and they think “if LLMs can do that, they can do anything”, because they see “easy” programming problems as hard. They have no concept of actual complexity, algorithmic or architecturally. They have an LLM ingest requirements that any developer with 2-3 years of experience would eye-roll at and then sneeze out a solution for in a week. The LLM can output something vaguely appropriate it in an afternoon, and they think that applies to any problem.
It does not.
LLMs are trained on existing code. If existing code could solve your problems, you’d probably be using it, instead of having your own developed. Existing code also has bugs. Lots of bugs. Bugs others haven’t discovered yet. Possibly even security bugs others haven’t discovered yet! And since LLMs are using existing code to “generate” this code, you get all the bugs from the existing code for free.
Remember also that LLMs take a long time to train, and models become out-of-date almost immediately. As one example: Recall Dirty Frag, one of the Linux CVEs from last month? I typed the first few lines of the function that had the issue into an unrelated C file, and Copilot’s autocomplete immediately suggested the vulnerable version:
This isn’t great for those relying on LLMs to write code for their business without review!
Trivialising the Humanity
My original analogy for this next section was going to be:
Writing yet another CMS vs writing a full-on novel software project is the difference between painting a wall and putting up a new wall. One needs a bit of artistic vision and a paintbrush; the other needs permits, training, and machinery.
I still like this analogy, at least enough to include it. But I’m not sure that even really covers it, though. Certainly the LLM can write you yet another CMS, however we’ve already seen it might introduce security issues (that might have already been fixed by what it stole from, ahem, trained on). And it misses something else: the humanity of it all.
Computer programming and software engineering, at least how I understand and practice it, straddle the line between engineering and art form. There is artistic expression in how you write code, the same way there is artistic expression in how you write prose.
Even if we admit that “yet another CMS” is not a great example of original expression, it still is expression. The same way someone writes corporate handbooks, the same way someone draws the little graphics that are shown on the product boxes that say “do not plug the iron in with wet paws” – someone is writing that CMS.
These sorts of projects are also great ways for people new to the profession to “get their paws dirty” without there being unacceptably high risk. I don’t know how else we can train people to write good things: they need to actually practice writing good things.
I’ve been writing articles since my grandmother inspired me to start back in 2002. I was 12 years old. Those early articles were actually pretty terrible! But I needed to write them, and practice writing, so that I could find my voice and unlock the talent I had. It is the same with programming.
LLMs could be useful after all, but only used responsibly
As I said at the beginning of this article, I think there are places where LLMs make sense. I think the vulnerability searches being done at scale are great and will make us (eventually) more secure. There is also high value in having different models doing other sorts of reviews; everything from accessibility to readability to finding design patterns in use that can be improved.
I see a lot of promise in machine translation coming out of these models. Personally, I also find it exciting to use them to search through collections of academic journals looking for the article I want to read. Gemini found me articles on canine osteosarcoma treatments that I simply could not pull out of WorldCat or Google Scholar. It’s amazing technology, when applied correctly and responsibly. Part of that comes down to model scale as well.
Honestly? If you ask me, I don’t think we need models as large as the industry is developing. Models at 1/8th or even smaller of the present size could solve these problems and provide us really cool search engines and translators. And I know that partially because that’s exactly what Apple are doing with their on-device models powering translation and search on iPhone.
In conclusion
Tech companies, and others, are infatuated with what LLM companies are promising. At the end of the day, what they are promising doesn’t wholly exist – at least, not in the way it is being sold. LLMs cannot take a novel idea and execute on it. LLMs can introduce bugs, security vulnerabilities, and regressions into your code the same way they can introduce new features and fixes.
Used responsibly, with human guidance, review, and intervention, they can be forces for good. They might even be force multipliers at some point. Yet, even the state of the art right now isn’t really capable of replacing a single engineer. And I don’t believe that the way LLMs work is compatible with ever replacing an engineer, though that gets into levels of math and neuroscience that aren’t really appropriate for a general-audience blog!
Ask not for whom the AI bell tolls. It doesn’t toll for you. LLMs will make us more productive, and might break down barriers someday. But right now, the industry is riding a terrible hype storm that will eventually crash down. And if I can give any advice, I’d say that I fear that economic crash far more than fearing being replaced by a machine.