Incident Report: The Angle Drifted

On 2026-03-09 the production diagonal was observed at 44.9997°. This is our public postmortem. It is blameless in every respect except one.

Summary

At 02:14 UTC on 2026-03-09, an automated check reported the production diagonal at 44.9997 degrees. The expected value is 45.0000 degrees. The drift of 0.0003 degrees was sustained for 41 hours before mitigation.

We are publishing this in full because we believe the industry has grown far too comfortable with orientations it has not personally verified this week.

Timeline (all times UTC)

02:14 — Detection. The angle monitor fired. The alert routed correctly. The on-call engineer acknowledged in four minutes, which is inside our SLO and the only part of this incident we are proud of.

02:31 — First response. On-call attempted to confirm the reading through the dashboard. The dashboard reported 45.0000 degrees. The dashboard rounds to four significant figures. The dashboard has rounded to four significant figures since it was built, and every angle it has ever displayed has therefore been unfalsifiable.

03:40 — Escalation. The Director of Angle was paged at home. He asked one question — "Have you looked at it with the protractor?" — and this question turned out to be the entire incident.

04:55 — Physical verification. On-call drove to the office. The protractor confirmed drift. The protractor is the only instrument we own that does not round.

09:00 → 18:00 — Investigation. Nine hours spent on the hypothesis that a recent deploy had introduced a floating-point error in the band calculation. It had not. The band calculation was correct. The band calculation has always been correct. We rewrote it twice anyway, which is a thing teams do at hour six.

2026-03-10, 11:20 — Root cause identified. See below.

2026-03-10, 19:26 — Mitigation applied. Angle restored to 45.0000°. Verified with the protractor. Verified again by a second engineer with the same protractor, which we now understand is not independent verification.

Root cause

The building settled.

The office is on the third floor of a converted brick warehouse constructed in 1911. Over the preceding winter, the northeast corner of the structure settled by an amount our landlord characterized in writing as "well within tolerance" and "extremely normal for a building of this age."

The rack holding the diagonal store is bolted to the floor in that corner.

The floor moved. The rack moved with it. The diagonal, which is defined relative to the rack, moved with the rack.

We want to be precise about the mechanism here, because it is the most important thing we have learned all year: the data was never wrong. Every value was correct. Every band was computed to spec. The software behaved perfectly throughout. What changed was the physical orientation of the frame of reference, and no amount of correct software can defend an orientation against masonry.

This is what we mean when we say the angle is a property of the observer. We had been saying it as a philosophical position. It is an operational one.

Mitigation

Four aluminum shims, placed under the northeast rack foot.

We considered a software correction — applying a +0.0003° offset in the band calculation — and rejected it. A software correction would have made the readings look right while the rack stayed wrong. That is not observability. That is a dashboard rounding to four significant figures, which is how we got here.

Action items

1. Replace the dashboard rounding with full precision. Done. It now reports eight decimal places and it is deeply upsetting to look at.
2. Add a laser inclinometer to the rack. Done. It cost $340 and it is the second-most-trusted instrument in the building.
3. Buy a second protractor. Done. Two protractors, two engineers, genuinely independent verification.
4. Structural survey of the northeast corner. In progress. Our landlord has stopped replying to email.
5. Move the rack away from the northeast corner. Rejected. The rack is where the rack is. Relocating it would introduce an unverified orientation, which is the precise condition this incident exists to warn against.

What we would tell you

Somewhere in your infrastructure there is a number you have never independently verified, displayed on a dashboard that rounds, in a building you do not own, resting on ground that moves.

Ours was the angle. We found ours. Have you found yours?