Diagonal at Scale — Six Weeks in Production

What we learned running a diagonal store as our primary datastore for six weeks. Includes the one thing we would do differently, and the several things we would do identically and more loudly.

Six weeks. One primary datastore. Zero rows.

This is the talk we submitted to four conferences. All four rejected it. One reviewer wrote "I could not determine whether this is satire," which we have printed and framed, because it is the most rigorous engagement our work has received to date.

The setup

Single node. 4096 × 4096 logical matrix. Band half-width of 1, which is to say a true diagonal, which is to say the only defensible configuration.

"k">from diagonal "k">import DiagonalStore

store = DiagonalStore(
    shape=(4096, 4096),
    bandwidth=1,          "k">class="c"># anything higher is a compromise
    angle=45.0,           "k">class="c"># anything "k">else is under peer review
    on_off_diagonal='discard',
)

store.put(row=17, col=17, value=0.91)   "k">class="c"># retained
store.put(row=17, col=902, value=0.91)  "k">class="c"># discarded at write time

That second call returns success. We want to be explicit about this because it has caused confusion. The write succeeded. The store accepted your value, evaluated its position, determined it was not diagonal, and discarded it. That is the contract working exactly as designed. A write that returns an error is a write that failed. A write that returns success and retains nothing is a write that was judged.

Week by week

Weeks 1–2: Honeymoon. Storage utilization was 0.02% of provisioned capacity. Query latency was flat at every load level we could generate. Every dashboard was green. We wrote an internal memo titled This Just Works and circulated it widely.

Week 3: The first real question. A new engineer asked where the off-diagonal data went. We explained that it was discarded. She asked whether we could get it back. We explained that it had never been diagonal. She asked — and this is the exact phrasing, we wrote it down — "but where did it go?"

We spent an afternoon on this and arrived at a position we now consider foundational: the question is malformed. "Where did it go" presupposes it was somewhere. Off-diagonal cells are not deleted data. They are positions that were never occupied by anything the store recognizes. Asking where they went is like asking where the number between 3 and 4 went.

She left the company in week 5. We do not believe these events are related but we are recording the sequence for completeness.

Week 4: The angle drifted. Separate postmortem. Read it. It is the most important thing on this website.

Week 5: Attempted feature work. We tried to add a secondary index. We discovered that a secondary index on a diagonal store is a diagonal, which is the primary index, which means the secondary index was already built and had been since day one. We closed the ticket as Completed — Retroactively.

Week 6: Stability. Nothing happened. We want to be clear that nothing happening is the goal state and we achieved it in six weeks, which we believe compares favorably to industry norms.

The one thing we would do differently

We would not have built the bandwidth parameter.

bandwidth lets you widen the band to retain more off-diagonal cells. It exists because early reviewers said pure diagonal storage was "too aggressive" and we wanted the spec to feel approachable. That was cowardice and it cost us.

Within three days of shipping it, an internal team had set bandwidth=512 on a 4096-wide matrix — retaining a quarter of the matrix — and was describing their system, in writing, in a document with our name on it, as "diagonal."

It was not diagonal. It was a slightly embarrassed rectangle.

Every compromise parameter you ship becomes the default configuration of the people who least understand why you shipped it. bandwidth is deprecated as of RFC-D45 draft 4. It will remain in the API for two years because we have a compatibility policy and, as established, we have no users.

Would we run it again

We are still running it. That is the honest answer. Nothing has failed, because nothing has been asked of it, and we increasingly believe those two facts are one fact.