Human comprehension is still the bottleneck
I've been using AI a lot more recently, and I genuinely think it has made me a better developer.
It has made me realise how much time I have been spending in the past just getting syntax right, or just doing the mechanical parts of coding, or the 'pretty-simple-but-still-requiring-concerted-thinking' mathsy or algorithmic parts of coding.
AI gets that stuff done that much faster, that it allows me to be thinking about the higher level patterns, and checking that they are right.
It means that I'm spending more time creating rules that enforce the patterns, than simply following the patterns myself.
It allows me to discover (via asking AI for suggestions) established solutions to a problem I'm facing, and then experiment with them, without fully understanding them, to see if they fit my usecase.
For example: my side project is a geometric art application. I have a facet that takes a series of points, and connects them via a line. But the low granularity of the points means we get jagged lines. AI was able to explain to me what bezier curves are (something I've heard of but not known what they are), but also suggested a Catmull-Rom spline for smoothing, which, although it's going to use a bezier curve under the hood, works for my usecase, because I don't need to determine the bezier anchors myself.
It's worth mentioning that there's possibly a bit of Dunning-Kruger at play here.
For example, I'm an experienced React developer, I know that AI isn't actually that good at writing React code, for example it tends to abuse useEffect a bit and it can unnecessarily make components stateful. Some of this might just be my personal preference. I have experience enough to recognise these anti-patterns and curtail them.
Whereas, where I now see AI as allowing me to learn in these other areas, it's possible that I have just enough knowledge to make me think that I/it is doing a good job of it, and not enough experience for me to recognise some well established pitfalls.
All this to say that - while AI helps move the developer into a space where they're thinking about the things that actually matter, and AI helps distill information to just what the developer needs to be informed about a thing. (To be clear, I mean informing developers about well established software engineering practices. At this point I would not trust an AI to tell me with a great level of accuracy the nuances of how a given codebase works.)
However - the limiting factor is still whether the human can understand what is happening.
Unless you are YOLOing it and creating a codebase that is not read by humans at all, you need to understand how the application works generally, how it is all put together.
So let's talk about code that has never been read by humans.
First, let me be very clear - there absolutely is a time and place to do this.
Creating throwaway prototypes is one such use case, where either myself, to test whether a concept is possible, or a non-coding product manager type to create a proof of concept of whatever idea they want to build.
I have found AI tends to be pretty good at these early stages, and it's as the codebase grows that things would run into the mud. The exact same problem exists for purely humans developing a codebase.
But beyond throwaway prototypes, it's plausible to have sections of the application that have never been read by humans. I can give several examples from my geometric art application.
Expand to see code
I have not read the these typings:
Lines 141 to 156 in fdac9be
141export type ConstrainedNodeInputsDeclared<
142 K extends keyof typeof nodeInputs,
143 Acc extends NodeAccumulator
144> = {
145 [Port in keyof typeof nodeInputs[K]]?:
146 typeof nodeInputs[K][Port] extends { valueType: infer VT extends ValueTypeNamesSuffixed }
147 ? K extends ControlNodeKinds
148 ? StaticValueDeclared<VT extends `${infer Kind}Value` ? Kind : never>
149 : ConstrainedValueDeclared<
150 VT extends `${infer Kind}Value` ? Kind extends ValueTypeNames ? Kind : never : never,
151 K,
152 Port,
153 Acc
154 >
155 : never
156 }Lines 54 to 75 in fdac9be
54
55type ConstrainedModuleNode<K extends ModuleNodeKinds, Id extends string, Acc extends NodeAccumulator> = {
56 id: Id;
57 type: K;
58 params: ConstrainedNodeInputsDeclared<K, Acc>;
59};
60
61interface ControlStageBuilder<Acc extends NodeAccumulator> {
62 addControlNode<K extends ControlNodeKinds, Id extends string>(
63 node: ConstrainedControlNode<K, Id, Acc>
64 ): ControlStageBuilder<Acc | { nodeType: K; nodeId: Id }>;
65 addComputeNode<K extends ComputeNodeKinds, Id extends string>(
66 node: ConstrainedComputeNode<K, Id, Acc>
67 ): ComputeStageBuilder<Acc | { nodeType: K; nodeId: Id }>;
68 addRenderNode<K extends RenderNodeKinds, Id extends string>(
69 node: ConstrainedRenderNode<K, Id, Acc>
70 ): RenderStageBuilder<Acc | { nodeType: K; nodeId: Id }>;
71 addModuleNode<K extends ModuleNodeKinds, Id extends string>(
72 node: ConstrainedModuleNode<K, Id, Acc>
73 ): ControlStageBuilder<Acc | { nodeType: K; nodeId: Id }>;
74 construct(): GeoArtGraph;
75}However, I did pay attention when doing TDD to create them:
Lines 741 to 753 in fdac9be
741 it("accepts a ref to a matching output port", () => {
742 // slider outputs numberValue; orbit.radius expects numberValue
743 assertType<ConstrainedNodeInputsDeclared<"orbit", WithSlider>>({
744 radius: { ref: "s1.value" },
745 });
746 });
747
748 it("rejects a ref to an unknown node id", () => {
749 assertType<ConstrainedNodeInputsDeclared<"orbit", WithSlider>>({
750 // @ts-expect-error - "xyz" is not a declared node id
751 radius: { ref: "xyz.value" },
752 });
753 });Lines 597 to 624 in fdac9be
597 it("Module node with invalid ref to module output port", () => {
598 const result = new AlgorithmBuilder()
599 .addModuleNode({
600 id: 'innerModule',
601 type: 'orbit-module',
602 params: {
603 time: { v: 0 },
604 speed: { v: 1 },
605 radius: { v: 0.1 },
606 numPoints: { v: 50 },
607 },
608 })
609 .addModuleNode({
610 id: 'outerModule',
611 type: 'orbit-module',
612 params: {
613 time: { v: 0 },
614 speed: { v: 1 },
615 radius: { v: 0.1 },
616 numPoints: { v: 50 },
617 //@ts-expect-error - 'innerModule.invalidPort' is not a valid port
618 centerPoints: { ref: 'innerModule.invalidPort' },
619 },
620 })
621 .construct();
622
623 expect(validateGeoArtGraph(result)).toBe(true)
624 })Also, for what it's worth, I have implemented typings like this by hand before.
I have not been reading this code:
161 const segStart = cumulativeLengths[seg];
162 const segLen = segmentLengths[seg];
163 const localT = segLen === 0 ? 0 : (target - segStart) / segLen;
164
165 const p0 = pts[Math.max(seg - 1, 0)];
166 const p1 = pts[seg];
167 const p2 = pts[Math.min(seg + 1, pts.length - 1)];
168 const p3 = pts[Math.min(seg + 2, pts.length - 1)];
169
170 const x = catmullRom(p0.x, p1.x, p2.x, p3.x, localT);
171 const y = catmullRom(p0.y, p1.y, p2.y, p3.y, localT);
172 const tanX = catmullRomTangent(p0.x, p1.x, p2.x, p3.x, localT);
173 const tanY = catmullRomTangent(p0.y, p1.y, p2.y, p3.y, localT);
174
175 // Colour: lerp linearly between p1 and p2 (the two nearest control points)
176 const colorT = localT;
177 const r = lerpChannel(p1.r, p2.r, colorT);
178 const g = lerpChannel(p1.g, p2.g, colorT);
179 const b = lerpChannel(p1.b, p2.b, colorT);
180 const a = lerpChannel(p1.a, p2.a, colorT);
181
182 return { x, y, r, g, b, a, dx: tanX, dy: tanY };Verfication is done by observing the behaviour of the application.
Snapshot tests detect if behaviour is changing.
This component was vibe coded without any attention to the code:
I did write a test though:
7test('renders a card for each node', async () => {
8 await render(<GraphView graph={minimalThreeNodeGraph} />);
9
10 expect(page.getByTestId('node-card-radius')).toBeDefined();
11 expect(page.getByTestId('node-card-time')).toBeDefined();
12 expect(page.getByTestId('node-card-earthOrbit')).toBeDefined();
13 expect(page.getByTestId('node-card-circle')).toBeDefined();
14});
15
16test('renders edges for each connection', async () => {
17 await render(<GraphView graph={minimalThreeNodeGraph} />);
18
19 // time.time → earthOrbit
20 expect(page.getByTestId('edge-time-earthOrbit')).toBeDefined();
21 // radius.value → earthOrbit
22 expect(page.getByTestId('edge-radius-earthOrbit')).toBeDefined();
23 // earthOrbit.points → circle
24 expect(page.getByTestId('edge-earthOrbit-circle')).toBeDefined();
25});
26
27
28The key thing to highlight here is that how these things are used is understood.
We might not have read the implementation - but we've read and understood the tests.
This strategy of 'encapsulate the implementation and expose the interface' ('don't worry about understanding the implementation, just understand what it meant to do') can exist at any level, whether you understand the function signature, but not the implementation, or it's an entire microservice whose REST interface you understand.
Could it be an entire application?
There's that video that went viral about Fable one-shotting a horror video game. Set aside my skepticism about the authenticity of it (ie. if the initial prompt has been carefully tailored and had multiple attempts to get right in order to 'one-shot', it's not exactly a true one shot), I think a video game of this genre, or any entertainment media really, can fall into the category of an application that doesn't need to be understood by the human, because it's an experiential thing.
Maybe the experience sucks but of course with any of these entertainment mediums, what's good is pretty subjective.
What I'm getting at here is, imagine a world where we can ask the AI to generate a game like Goat Simulator - and it goes off and does it, the code compiles, the game doesn't crash, we never read a single line of code.
I think this is the most realistic scenario for 'an application that humans don't need to understand'.
If on the other hand the game was something like Civilisation or Europa Universalis - these are games that 'the human needs to understand' - the game has rules and we need to make sure the rules are being applied consistently, otherwise that's not fun.
An experiment - designing a video game
I started an experiment where I was going to create a video game, I've never created a video game before, so this would be building something in a programming language and framework that I had zero familiarity with.
The game was going to be like a third-person WASD Factorio-like game where you could be a gold miner initially panning for gold, moving up to blasting and tunnel building, or maybe you run a general store, or maybe you're a railroad baron.
So I started doing ideation sessions with the AI, just creating markdown files, planning out interactions, and what the game objects that the player could build etc.
I got about an hour and a half into this, before I realised 'this is a lot of work' - before even a single line of code has been written.
So AI isn't going to be able to build me this game in a weekend. It might be able to build a game, but it won't be the one that is in my head.
It's possible that in some future, it will be able to build the thing that is in my head. If the AI has enough understanding of what's in my head, whether that's because of a brain implant, or just because it knows enough about human psychology generally, that with a limited prompt and short back and forth it can infer just what makes me tick.
There's also the possibility of the future of games and applications generally being tailored to the individual user. Particularly with a single player game, you can imagine an in-game prompting window where the player can say 'this is good, but it would be cool if I could fly/occasionally monsters were waiting around the corner/the view was first person only' and the game is adjusted for them. Easy to see how this would work for single player games, multiplayer is less clear.
The bottleneck is still human comprehension
Where this comes around to, is that I don't quite understand how people are using huge numbers of tokens. I see people mentioning that they've got multiple things happening simultaneously, and they're just managing the AI. But in order to know what to tell the AI what to do, you have to understand something about the domain. What is this world where people already understand everything that needs to be done?
This actually gives me a bit of pause, there's a sense that AI is going to make the job of software developers harder because there's less routine grind work to give yourself a break, it's all high level conceptual understanding.
But still, what things can we give up understanding?
I don't need to know exactly how a car works in order to drive one. If the car is cheap enough, I don't need someone else to understand it either, if it breaks, I can just get a new one. Of course, we do want to make sure the car is safe.
It occurs to me that insistence on knowing what the AI is doing, would be like a programmer in the era where compilers were new, insisting that you read the compiled machine code. This analogy isn't quite fair though - compilers are deterministic, LLMs are not. On the other hand, at a sufficient level of complexity, does it matter if the thing is deterministic or not? A deterministic black box of mystery is still a black box of mystery.
A different analogy is that the CTO of a tech company doesn't understand every line of code in the company, they trust their subordinates who in turn trust their subordinates. Perhaps it comes down to responsibility - if I'm the CTO and someone below me completely breaks something, some investigation is done, and either someone below me gets fired, or I do. Somebody is ultimately responsible for the screw up.
If I'm a vibe coder... there's no one below me to take responsibility - the LLM isn't going to. So it behooves me to at least understand the potential risks of the application that I'm building-but-not-understanding.
Questions? Comments? Criticisms? Get in the comments! 👇
Spotted an error? Edit this page with Github