WIP

This is a catch-all post expanding on a couple of points about LLMs that I’ve been meaning to jot down over the last few months, based on my limited experience working with them and the general sense of AI discourse I get from Twitter. My main goal is condensing these thoughts into some kind of coherent thesis-ramble. These points sound aggressively confident, so please read them critically and with a pinch of salt.

1

We still are not sure who’ll capture the value in AI.

I really like this piece ¹ in the Colossus magazine, where Mr. Neumann makes the case that like containerisation (shipping not software) capturing net-value is unlikely even if AI is very useful to consumers. The best case scenario for looking for good and valuable opportunities in AI seems to be “think through the implications of knowledge workers becoming more efficient, to imagine what markets this efficiency unlocks, and to invest in those” (from the article).

In terms of AI products, I’m surprised that I only interact and get value out of a couple of AI products - primarily claude through claude.ai, Zed text threads and claude code. In terms of value addition for customers in AI products (a necessary but not sufficient condition for value capture), the bare minimum test is that the customer should not be able to navigate to chatgpt and perform the functionality with basic functionality. If that is the case, the product, there isn’t much of a moat to start off an the customer is almost always going to prefer chatgpt for its flexibility and the switching cost is not worth it.

For me personally, i speculate some of the characteristics of a valuable AI product are :-

• Outcome: Solves specific problems E2E deeply (eg. automated pentesting, automated BI dashboards).
• Speed: Eliminates waiting states by removing bottlenecks (eg. automated code review)
• Bring the AI to me, don’t take me to it: This is more of an characteristic in incumbents (eg. AI suggestions in Gmail and Docs, AI Overview in Google Search)
• Good UX ²: A personal favoutite of mine. (eg. text prediction in Cursor, an agent interface in claude code, voice-to-text, meeting notes, notebook LM)
• Cost: Decrease cost, especially labour-related (eg. AI customer support, document review)

---

2

• Diffusing AI into the economy isn’t as cut-and-dry as its made out to be, it’s a lot of work to just get it reliably deployed in a valuable production setting.
• Models performance seem to be stagnating in 2025 (at least not showing as much of a rapid improvement as before ? ).

I’ll try to keep this brief. The first point follows partly from the second point. I think people vastly overestimated how easy it is to get any serious AI system deployed in enterprises or into anything important³. There’s even an MIT study that shows 95% of AI pilot projects failing⁴.

But I strongly believe that most companies are not even scratching the surface in leveraging existing AI capabilities (especially for internal business processes and operations). Like most issues I get the sense that diffusion is a socio-technical issue with inherent friction. Still there’s enterprises leveraging AI properly out there⁵.

Also I don’t think AGI (or atleast what most people consider AGI) is in direct sight and a more relevant metric to understand value of AI systems is in its diffusion.

---

3

There’s clearly a marked impact on knowledge work ( especially software engineering) .

Personally I’ve found LLMs to be clearly useful (see AI_UI_UX & claude_code_usage_cost ). I started playing around with LLMs late last year, I started off with using the Zed text threads, basically adding a file (or chunks of it ) context into a thread, talking to an LLM about it and then pasting the generated code back into my codebase. A couple of months later I started using claude code to implement features and fix bugs in isolated modules and that was fun too (except i was bored waiting for claude code to complete a task).

Over the past couple of months, I’ve also encountered some scenarios like reviewed “vibe-coded” PRs that are so horrendous and non-sensical - that degraded the codebases to a big ball of mud-slop in a couple of months. Another such scenario was when claude code spent a lot of time implementing a feature , added some bugs, was never able to recover from it and then gave up; which lead me to spend a lot of time trying to just understand the code and then some fixing it, when it would have taken me way less time just implementing it from scratch. My sense is this stems from two issues. One is a constraint of the models that it cannot maintain a clear mental model⁶. And two general socio-technical / quality-bar-to-low / bad-llm-usage problem (eg. team not investing in quality gates for AI-generated code).

I’m not too concerned about people becoming insignificant in the software development process because it’s still not clear to me what kind of workflows will get us to a state of LLM generated code being deployed all the way till production with minimal human intervention. A person still has to take responsibilty for any serious software system and its evolution in production. Software engineering consists of more than just code generation.

A second order effect I’m interested is the effect on quality of software systems. In my time working, I’ve learnt that the quality of software is more or less dictated by finance and economics more than anything and most code is already pretty terrible. But I do expect at least a few teams to leverage LLMs to get to better software quality. ⁷

Another concern I’m interested is folks getting lazier and not having a good understanding of their systems that have to maintain. Generally “Not having a good understanding of systems” is more a symptom of a human fallability (of lack of ownership) and organisation incentives at play. In my opinion laziness is a bigger concern since using LLMs extensively leads to you not exercising some skill muscles and loosing them ⁸. Another aspect of the laziness is folks loosing the researching nerve, and blindly trusting LLMs not considering stuff like confirmation bias.

---