You’re reading this on a website I didn’t write. Not a line of the code behind it — the layout, the styling, the build all came out of an AI. It compiles, it runs, it looks great.
It also makes almost no decisions. A marketing site is presentation, not logic; there is nothing here to get quietly wrong. Letting a model run free on it was exactly the right call.
Now ask that same model to write code that prices a bond or checks a position against a regulatory limit.
Finance is the opposite of a brochure
It’s all logic, and the logic is treacherous. A price is always positive — until the April day in 2020 when WTI oil futures traded at a negative price, reaching a low of -$40 a barrel. A trade settles T+2 — except in markets that don’t, or when a holiday pushes it out. One ISIN identifies a security — except that it may trade in different places, at different prices, in different currencies. Every “always” may hide a “sometimes”.
That is where a coding model gets lost. It learns the rule it has seen ten thousand times — a price is positive — and quietly encodes it everywhere: a validation that rejects
negatives, a type that cannot hold them, an abs() that “cleans up” the data. The logic is plausible, but it is catastrophically wrong.
This is not hypothetical. When oil went negative, Interactive Brokers — an established broker with decades-old systems — found its own software could not cope. The negative numbers broke the model that calculated customer margin, and the firm paid $82.57 million in restitution to customers and a $1.75 million penalty. The assumption that a price is positive had been wired in for years. Nobody had noticed — because until that afternoon, it had always been true.
Plausible looks exactly like competent
That is the trap. Bad AI code does not look bad — it looks like what a good engineer would write. The bugs are the quiet kind: an off-by-one in a date window, a nullable treated as if it can never be null, a missing edge case. No compilation error, no failing build, no stack trace.
In a consumer app, that’s survivable. A subtle bug is a broken button; you ship, someone complains, you patch it. “Move fast and break things” exists because in many situations, breaking things is cheap.
In a trading or risk system it is not. A quietly wrong number is not an annoyance — it is a position someone took by mistake, a limit reported as checked that was not, a trading error or even a fine. You cannot un-trade a trade.
It is not the model’s fault
None of this means AI is overhyped, or that a specific model is bad. It means we are asking the wrong thing of it. Models are trained to be agreeable and to produce plausible output — that is the job. Expecting one to police its own way to correctness, in a domain built entirely of exceptions, is like handing a brilliant, fast, slightly overconfident junior the keys to production with no tests, no review and no architecture — and expecting them never to make a mistake.
The productivity is real. It just does not come from the model running free. It comes from what you build around it — the gates it cannot argue with, the process it has to follow, the review it cannot charm, and a human who keeps final authority over anything that touches critical processes such as trading or risk. In the next articles, we will explain how we capture that productivity by keeping the model on a short leash.