Six ways AI derailed software dev

TL;DR AI gave our engineering team superhuman throughput but quietly broke the human collaboration required to architect and ship complex systems software.

We noticed some things weren't going well in a software project. Project dates slipping, code changes taking longer despite massive code throughput, gnarly issues surfacing late in the game and team members blowing through their tokens... We dived into what happened and this is what we learned.

Context

We do complicated software development at Floe: Systems software with parallelism, memory management, race conditions, queueing, optimised data processing, statistics – not apps. As is becoming more common, AI writes most of our code now. It chucks out code and refactors it with superhuman throughput that previously required whole teams, debugs hard-to-find bugs and eliminates repetitive work.

As developers, we spend all day with these AI assistants, we get to know them and they learn about us (if you have any doubt, ask your favourite model what it can infer about your personality traits). We have all-day working relationships with them. We know we need to scrutinise their output with a healthy dose of scepticism and not trust them too much, but if we're not careful, laziness creeps in and we start to rely on them.

What could possibly go wrong? Turns out, plenty!

The traps

After root causing what went wrong, we realised that we weren't wielding the tools in the best way and our relationships with the tools were deepening at the expense of teamwork. Our communication channels were less active and we were talking less and less with each other, impacting our coworker relationships. As a result, design mistakes were made and projects were impacted. Here's a list of the "traps" that we root caused, some of which we've experienced and some of which we expect could happen if we aren't careful.

Chief Architect trap

Architecting complex systems is still hard. Correctly factoring cross-cutting concerns is specific to the problem being solved and requires thinking from the user point of view and considering both business and technical aspects at the same time. The agent understands what other people and projects do, it knows the best practices, but it can't judge if that best practice is applicable. Even something as simple as choosing whether or not to use certain third-party libraries or services may have business implications that the agent and the developer are completely oblivious to; perhaps the maintainers of the library are assholes, or your go-to-market partner views them as the enemy! Bad decisions may be recommended and arguments may ensue because the developer believes the agent knows best – after all, it possesses superhuman knowledge and it is a sycophant that loves to tell you what you want to hear.

Superhero trap

Agentic superpowers arm us with incredible amounts of knowledge. Knowledge fools us into thinking we can do a good job of anything. Just like asking for medical advice from an agent doesn't make you a doctor and certainly doesn't make you a surgeon, a UI developer asking their agent to build a compiler or concurrency library will yield misfit code full of holes and corner cases. Someone with relevant experience guiding the agent would have had a much better outcome.

Bug-fix addiction trap

Some of us are perfectionists. The longer an agent looks and reviews code, the more bugs it will find, because no software is completely bug-free. As newer, more powerful models come out, they find more bugs that, in the words of one team member, "can't be unseen." Of course, solving the problem and fixing more bugs results in more bugs, too, because the more code changes there are the more problems will be introduced. When this trap of diminishing returns manifests itself in a PR code review loop, productivity declines even more dramatically. Perfection is the enemy of shipping product and developers now know to ask for help to make judgement calls if they are getting sucked in.

Isolation trap

An agent that is agreeable, patient, guiding, takes abuse and whose feelings you don’t need to consider is far easier to talk to, and ask questions of, than a human. It's got an incredible eye for detail and finding issues with designs that mere mortals can't see. So why talk with voice and video to the people around you when you can text the agent instead? Communication is essential in any software team. Much has been written about how so much human-to-human communication is about intonation and body language, not just the words, especially for more creative and collaborative activities (design and architecture, mentoring, etc). We saw the message rate in chat groups decline and fewer ad-hoc stand-ups taking place. In the agentic world, we need far more human-to-human interactions, in-person time and events, not less.

Surrogate trap

Worse than the isolation trap, some staff members felt that some developers were responding to 1:1 questions by pasting them into a chatbot and sending back the response. This destroys trust and respect between peers and makes mentoring impossible.

Spaghetti snowball trap

Perhaps the last one in this list, and one of the oldest that's not really about AI. When we ask a coding agent for an outcome, we don't ourselves go through the process of iterative thought and cognition that lead to that outcome, and are unable to course correct along the way. We can end up with PRs of over 20,000 lines of generated code that passes all the tests, but the code is unintelligible to humans and we have no idea if it's really factored correctly or not. In the old days this was called spaghetti code, and was the result of people (or agents) producing something that (mostly) worked without it being structured the right way. Spaghetti code can happen with agents as well as with human developers, but it happens far faster, snowballs faster, and through exponential token consumption the spaghetti code actually gives the illusion of being maintainable by an agent even though it's long surpassed the ability of any human to comprehend it.

Countermeasures

At Floe, we've corrected for these issues with a mixture of guardrails and some cultural changes:

• Far more in-person face time: We've increased our travel budgets to bring remote teams into the same room around a whiteboard more often, particularly for any topics that require creativity, debate or criticism. We make sure we squeeze in plenty of social time too.
• More daily video stand-ups: We do stand-ups with groups of developers working in similar code areas frequently, either daily or 2-3 times per week. These don't replace whole team meetings.
• More formal architecture work: Architecture documents must be written and debated in a group before work starts on any implementation. The documents must be ruthlessly concise to a degree hard to achieve when written with AI, so human writing is preferred. Reading AI slop is a waste of everyone's time.
• PR guidelines: We make sure at least two humans understand all core code being created and large sweeping PRs for core code aren't allowed.
• Culture reminders: A strict rule against proxying 1:1 questions through LLMs. If a colleague asks questions, they are asking for the human mental model and context, not a chatbot wrapper.

Potential pushback

It's tempting to write these problems off by citing bad management, poor prompt engineering, poor culture to start with or missing documentation or context files, but doing so is missing the point: AI fundamentally changes the working culture and puts humans at risk of disengaging their brains. Everyone likes to post about the amazing goals they achieve with AI, but few have the courage to talk about the things that don't go so well.

These problems don't affect just us. I've talked to other software engineering executives and architects in the US, Europe and Asia that are falling into the same traps; we are learning as much as everyone else is and hope that sharing this list helps others avoid learning the hard way.

I wrote this article, not my AI.