DIAGONAL DATASETS

Position paper · revision 6

The Thesis

Every data system ever built has offered you exactly two orientations, and it has offered them so consistently, for so long, that the offer has stopped registering as a choice.

Row-oriented. Column-oriented. Zero degrees. Ninety degrees.

This document argues that the restriction is arbitrary, that it is historically contingent rather than technically necessary, and that the third orientation has been sitting in the middle of your matrix the entire time being described as an edge case.

1. The restriction is not technical

There is no theorem that forbids a 45° storage layout. There is no paper establishing that only the axis-aligned orientations are tractable. We have looked. We have asked people who would know. The most common response is a pause, followed by some version of "I suppose there isn't, but nobody does that."

Nobody does that is not a finding. Nobody does that is a description of a field's attention, not of its constraints.

The honest history is that early storage hardware was addressed linearly, linear addressing made one axis cheap and the other expensive, and the entire discipline organized itself around that asymmetry. Then the hardware changed completely, several times, and the organizing assumption did not.

2. The transpose closed the question

We develop this fully in Against the Transpose, but the short form belongs here.

The transpose operator maps row-orientation to column-orientation at essentially zero cost. Because the two conventional layouts are freely interchangeable, the field concluded the choice between them was a tuning decision rather than a design one — and having decided the choice was minor, it stopped searching the space of choices altogether.

The transpose is a loop between two states. Almost nobody has asked what is outside the loop, because the loop is comfortable and it is one character wide.

What survives the loop unchanged is the diagonal. In the most-used operation in applied linear algebra, the diagonal is the fixed point, and it has been taught for sixty years as a footnote.

3. 45° is not one option among many

Of the infinitely many angles available, exactly one is self-inverse under transposition.

At 45°, your layout after a transpose is bit-for-bit identical to your layout before it. At every other angle θ, the transpose sends your band to 90°−θ, you now have two canonical forms, and you need a committee to choose between them.

This is why we are not angle-agnostic and never will be. 45° requires arithmetic. Every other angle requires politics. See 30° and 60° for the standing dissent, which we have recorded because a standard that does not record its dissent is just a memo.

4. The discard is the point

A diagonal store evaluates every write against the band. Writes inside the band are retained. Writes outside it are discarded at write time — permanently, with no recovery path and no archive.

This is the provision people cannot accept, and it is the only provision that matters.

A store that quietly keeps the off-diagonal cells "just in case" is a dense store wearing a costume. You will pay for the full rectangle, you will scan the full rectangle, and within a quarter you will have an incident whose root cause is a query that touched the part you were only keeping just in case.

The discard is not a side effect of the diagonal. The discard is the diagonal.

5. You already built this

The strongest evidence for the thesis is not in this document. It is in your infrastructure.

Look at your retention policies. Look at the TTL someone set in 2021 to make a bill smaller. Look at the feature store where the lookback window narrows as the feature index grows. Look at the correlation matrix your risk team truncates past a certain distance.

Those are bands. They are oriented, to a first approximation, at 45°. They were built under budget pressure by people who never named the shape they were building.

Our claim is not that you should adopt something new. Our claim is that you adopted it already, for the wrong reasons, and that naming it correctly is the difference between an architecture and a series of compromises that happen to rhyme.

6. What we are asking

  1. Recognize the diagonal as an orientation, not a subset.
  2. Stop treating the fixed point of the transpose as trivia.
  3. Read RFC-D45. Object to it in writing. Objections in writing are the only kind we can put in §7.1.
  4. Before your next schema review, ask one question out loud: what angle is this?

The question is not rhetorical. Someone in the room will have an answer and it will not be a good one.

The data was always there. We just needed the right angle.