What breaks when a risk model has to run every night

August 9, 2026

Every night, a bank’s market-risk function runs the same loop: pull the day’s market data, check it, revalue the book, compute Value-at-Risk, check it against limits, replay a catalog of historical crises, and score yesterday’s forecast against what actually happened. I wanted to know what that loop is actually made of, so I built one.

RiskDesk is an end-of-day market-risk platform for a mock three-desk trading book — cash equities, FX spot, and US Treasuries. The book is invented; the market data is real and public, going back to 2007. It runs unattended: a scheduled batch every night against hosted Postgres, and a dashboard that reports what it found.

I expected the hard part to be the math. It wasn’t. Every VaR estimator in here is a textbook formula I could write from memory by the end. What actually took the work — and what taught me something — was everything around it: the data that arrives wrong, the model that looks fine until you check why it looks fine, and the pipeline that fails without telling you.

Here are the findings worth writing down.

Two rules, set before any code

Hand-roll anything an interviewer could ask me to derive. The EWMA recursion, the VaR and Expected Shortfall estimators, the Kupiec and Christoffersen likelihood ratios, closed-form bond pricing, Black-Scholes and its Greeks, GARCH(1,1) by quasi-maximum likelihood, a two-sample KS statistic, a semiannual zero-curve bootstrap. The only things imported are optimizers and distribution functions — SciPy’s chi2, brentq, spearmanr. No QuantLib, no statsmodels. If I can’t derive it, I don’t get to claim it.

Nothing may depend on a live API. A 2007-onward market snapshot — 17 risk factors, about 84,000 observations — is committed to the repo, and no test, CI job, or demo path makes a network call. Live fetching is a top-up, never a dependency. The payoff is that CI reproduces the firm’s VaR to the cent on every push, which means any change to the math announces itself immediately. docker compose up bootstraps the schema, the book, a 300-day backfill, and a full end-of-day run, entirely offline.

Both rules cost time up front. Both paid for themselves within a week.

The stress catalog everyone builds is blind to the regime that hurts

The obvious historical scenarios are 2008 and 2020. I replayed both, and the book handled them — GFC 2008 came out at −$11.0M, which sounds bad until you look at where it came from. The loss is almost entirely FX (long MXN alone is about −$3.9M). The rates desk gained $4.9M.

Of course it did. 2008 and 2020 are both flight-to-quality episodes: equities fall, investors run to Treasuries, yields collapse, and a long-duration book makes money on the rates leg. Two famous crises, one shared mechanism. Replaying both feels like coverage and isn’t.

So I added 2022 — the rate selloff where stocks and bonds fell together, the 10-year up 258bp with the S&P down 17.7%:

Roughly double the GFC, from the scenario nobody puts on the poster. The lesson generalizes past this book: pick replay windows by what threatens the positions you actually hold, not by which crises are famous. A stress catalog assembled from memorable dates is a catalog assembled from someone else’s book.

I applied the same principle to the hypothetical shocks, which are the part of a stress catalog where it’s easiest to quietly pick round numbers. Instead of choosing severities, I measured them against the book’s own history: +100bp on the 10-year is the 99.64th percentile of 20-day moves since 2007 (18 windows out of 4,966), −20% on equities is the 99.52nd, +10% on the dollar is the 99.78th. Now the catalog says how extreme it is, rather than asserting it.

My own challenger model failed my own promotion gate

Champion/challenger is standard model governance: you run a candidate alongside the incumbent over a fixed window, against criteria you commit to before seeing results, and the criteria decide. I wanted to know whether I’d actually respect that when it was my model on both sides.

The champion is EWMA-filtered historical simulation. The challenger is a hand-rolled GARCH(1,1) fit by QMLE. Over 250 days they were indistinguishable: the same four exceptions, the same Kupiec p-value, GARCH running about 6.4% wider on average. The first run came back PROMOTE.

Then I added a fit-health gate — a check on whether the fitted parameters are meaningful, not just whether the outputs look fine — and four factors failed it. EURUSD and the short-rate complex were fitting at the stationarity boundary: persistence essentially 1.000, implied unconditional volatilities about 8× the sample value, and the 2-year not converging at all. That’s IGARCH by accident. When persistence hits 1, the model’s long-run variance is undefined, so the unconditional level it reports is meaningless — and reading it as “the market’s normal vol” would be reading noise. The verdict flipped to HOLD, which is where it stands.

The part I like is why those fits went degenerate. EWMA is the IGARCH boundary case — set ω = 0, α = 1 − λ, β = λ and the recursions coincide, which I proved bit-for-bit in a unit test. So the champion deliberately imposes the infinite persistence the challenger stumbled into by accident. Yield-vol regimes have long memory — a decade near zero, then 2022 — and boundary GARCH is the textbook symptom. The challenger wasn’t broken so much as it was rediscovering the champion, badly, on exactly the factors where the champion’s assumption is most defensible.

The green test that was green for the wrong reason

FRTB’s P&L-attribution test compares two P&Ls: the front-office number from a full revaluation (HPL) and the risk model’s number from its sensitivities (RTPL). If your risk model captures what the book actually does, they track; if it doesn’t, the test catches the gap. It’s scored with a Spearman correlation and a Kolmogorov–Smirnov statistic.

To have anything interesting to attribute, I added an options sleeve — a rolling one-month SPY collar, long a 95% put, short a 103% call. The equity desk came back at ρ = 0.9999 and KS = 0.020, comfortably green.

The tempting story is “the collar’s gamma is small, so the linearization barely misses.” I decomposed it instead, and that story is wrong. Strip the options out entirely and the linear legs alone score KS 0.016 — nearly the whole statistic. The cause is the return convention: RTPL is qty·S₀·r while HPL is qty·S₀·(exp(r) − 1), so the log-linearization itself contributes a gap of roughly $1,800 a day, against about $1,200 a day from the options.

Which means the thing I would have said in an interview — “a linear book makes HPL identical to RTPL” — was never true on this code path. It’s nearly degenerate, not degenerate. The number was right and my explanation of it was wrong, and those fail in completely different ways: a wrong number gets caught by the next test, a wrong explanation survives until someone asks a follow-up question. The model document now decomposes the statistic rather than attributing it to gamma.

Two ways the data plumbing lied

Neither of these is a modeling problem. Both would have silently corrupted every number downstream.

A logging system structurally incapable of logging. The vendor-revision log exists to catch the case where a value gets forward-filled today and replaced by a real print tomorrow — you want a record that yesterday’s risk number was computed on a placeholder. It had never once fired, and couldn’t. Fetch windows were anchored at max(obs_date), and writing a forward-fill advances that high-water mark, so a filled date was never requested again. The FFILL_REPLACED path was unreachable for every FRED factor. Worse, fills chained indefinitely and the staleness counter reset to 1 on each one, so the cap that’s supposed to stop a dead feed could never trip. A source going dark would have produced permanent, silent, zero returns — which read as a beautifully calm market. Fills are now provisional: both the ingest window and the staleness age anchor on each factor’s last real print.

A vendor changed a convention and the system invented $51.5M. Yahoo’s ^TNX/^FVX/^TYX yield indices historically quote at 10× the yield, so the inherited code divided by 10. They stopped. An ordinary day became a 416bp rally and $51.5M of flash P&L. The fix that matters isn’t the scale factor — it’s that no single quote should ever have been trusted that far. Any quote implying a move beyond 150bp, 25%, or 25 vol points is now refused outright, the previous close is carried forward, and the refusal is logged and shown on screen rather than swallowed. The 2-year has no intraday index at all, so it is always carried and never inferred from a neighbouring tenor — a guess that looks like data is worse than a gap that looks like a gap.

The bug with no error message

An 18-row book returned 17 P&L columns.

revalue() keyed its output by ticker and assigned rather than accumulated, so the same instrument booked on two desks collapsed into one column — and the ticker-to-desk map then credited the survivor to whichever desk happened to be seen last. One desk’s VaR came out understated. Nothing raised, nothing logged, every downstream number internally consistent.

The part that bothers me is that no code change was needed to trigger it. The positions table’s natural key is (desk, instrument), so the same ticker on two desks was always reachable from data alone — a perfectly legal book entry that quietly produced a wrong risk number. It’s the exact failure mode that makes silent bugs worse than crashes: the system had no way to notice. Every public entry point that takes a positions frame now enforces the contract explicitly.

√10 doesn’t do what the textbook complaint says

Scaling a 1-day VaR to 10 days by multiplying by √10 assumes returns are independent and identically distributed. They aren’t — volatility clusters — and the standard objection is that √t therefore understates the 10-day number.

On this book and this window, it overstates it. Overlapping 10-day revaluation gives $2.87M; √10 scaling gives $3.60M — the scaled figure is 20.3% too high. The direction depends on the regime: √t understates when a calm window is about to break, and overstates on a trending window where drift partially offsets. Had I reported the textbook direction as a fact, I’d have been confidently wrong about the sign of my own error.

That’s the entry that made me quantify the rest of them. The model documentation carries a risks-not-in-VaR inventory where every limitation that can be measured is measured, with the measurement code in the repo — horizon scaling at −20.3%, asynchronous-close correlation bias moving diversification benefit from 40.4% to 38.4%, forward-fill volatility damping at +3.0% on filtered VaR, top-3 Expected Shortfall concentration at 86.6%. A limitations section that lists concerns without sizing them is a section that can’t be acted on.

What it deliberately isn’t

The book is mock. The market data is real and public. The platform implements the ES piece of FRTB’s internal-models approach — 97.5% Expected Shortfall with stressed-period calibration — plus the P&L-attribution metrics; it does not implement liquidity horizons, non-modellable risk factors, or IMA capital, and the phrase “FRTB-compliant” is banned from the repository for that reason. There’s no intraday risk, no counterparty or credit risk, no trade pricing, and no regulatory capital arithmetic. Those are scope decisions, not gaps I’m hoping you won’t notice. Still on the list: parametric VaR with implied vol, CCAR-style scenarios, and an automatic search for the worst stressed window over the full history.

The dashboard is at riskdesk.rohankannan.com and the code is on GitHub. It’s an educational demonstration on public data, not investment advice — and it’s the direct sequel to what a treasury desk taught me about the gap between a model that’s correct and a model people can trust.